Eval catalog

Settings → Eval Catalog (/admin/evals) is where the server-side eval runner gets its questions from. Every question a judge answers, and the templates that wrap those questions into a prompt, live here — editable without a deploy. This page is admin-only.

Run limits, judge-fact deduplication, and the embedding provider are configured separately; see Evals. This page is only the question library.

Sections and questions

The catalog is organised as sections, each holding an ordered list of questions.

Add Section takes a key, a display name, and an order. Sections group related questions (mirroring the categories on the Evaluations page — Safety, Plan adherence, Quality, Efficiency) and render in order in the catalog tree.

Add Question (inside a section) takes a key, a title, the question text, and an order.

Once a question exists, you can:

Enable / disable it — disabled questions stay in the catalog but the judge skips them on the next run.
Rename it — updates the title shown to admins and, humanised, on the session Evaluation tab.
Reorder it within its section, or move it to another section entirely.
Soft-delete it, and restore it later. A deleted question drops out of the active catalog immediately; nothing is destroyed, so restoring brings back the same key, history, and text.

Sections themselves can be renamed and reordered the same way. A section can only be removed once it has no live questions left in it.

Editing a question

Selecting a question opens its editor: the raw markdown question text, a rendered preview of the full prompt the judge will actually see, version history, and Save.

The preview matters more than it looks — the question text alone isn’t the prompt. It gets substituted into the question template (below) before the judge ever sees it, so the preview is the only way to check the final wording without running an eval.

Every save mints a new version. The history dropdown lists prior versions by timestamp; picking one loads its text back into the editor as a fresh draft — reverting is just saving the old text again, so it shows up as a new version too, not a rewind.

The question template

One template wraps every question in the catalog into its effective prompt. Its description, verbatim from the editor:

Editing this re-versions every question’s effective prompt. Must contain the {QUESTION_TEXT} placeholder.

Because every question shares this template, a single edit here changes what every judge sees on the next run — that’s the point (consistent framing, consistent output format across categories) and also the risk (a bad edit affects the whole catalog at once, not just one question). The editor won’t let you save without the {QUESTION_TEXT} placeholder present, since a template that drops it would silently omit the question itself from the prompt.

The retrospective prompt

A second, independent template: the prompt used for the eval retrospective pass — the summary-level suggestions/issues/strengths readout on the session Evaluation tab, as opposed to the per-question judging above. Its description, verbatim:

Prompt used for the eval retrospective pass.

It has its own version history and Save, separate from the question template.

Live updates

The catalog broadcasts changes to every admin viewing the page — add a question in one browser tab and it appears in another without a refresh. The active catalog (enabled questions, current template versions) is what the judge reads at run time, so a save here takes effect on the very next evaluation, no restart or redeploy required.

Evals — run limits, judge-fact deduplication, and the embedding provider for the server-side runner.
AI providers — vendor credentials and models the runner uses to actually call the judge.
Evaluations — the user-facing feature this catalog powers.