Evals

Settings → Evals (/admin/settings/ai/evals) configures how the server-side LLM-as-judge runner behaves once vendor credentials are in place: how long a run is allowed to take, how much trace it sees, and whether judge findings get deduplicated with embeddings. Vendor API keys live on AI Providers — this page doesn’t hold credentials, only behavior.

Run limits

Per-run budgets for the server-side runner:

Key	Range	Default	Description
Evals:ServerRunner:RunTimeoutSeconds	60–3600	900 (15 min)	Wall-clock budget for an entire eval run. On expiry the run surfaces as EvalFailed("cancelled").
Evals:ServerRunner:MaxTraceChars	10000–2000000	400000 (~100k tokens)	Upper bound on the formatted trace passed to each judge prompt. When exceeded, the orchestrator keeps the most recent events and prepends a marker noting how many were elided.

Judge-fact deduplication

JudgeFactDedup:Enabled is the master switch for the semantic dedup pipeline:

• Off — eval-generated judge facts are retained verbatim, and no embeddings are computed. The Voyage / OpenAI embedding APIs are not called at all.
• On — candidate facts are embedded and compared against the existing pool; near-duplicates collapse into clusters. Toggling this on also triggers a background backfill of any facts that accumulated while dedup was off.

The active embedding provider’s API key must be set on AI Providers before you enable this.

Embedding provider

Embedding:Provider selects which provider runs the judge-fact semantic deduplication pipeline — required when judge-fact dedup is enabled. Choices are voyage, openai, and (on plans that allow it) kurrentai.

Saved changes take effect immediately; no restart is required. API keys and per-provider models live on AI Providers — this selector only decides which one is active.

Caution

Changing identity fields — the provider, its model, or (for OpenAI) the dimension count — discards all stored embeddings. The backfill service re-embeds retained judge facts under the new identity in the background. Until it finishes, judge-fact clustering has nothing to show.

The eval catalog

Evals also has a link out to the eval catalog — the set of evals the server-side runner can execute:

Edit the eval catalog → /admin/evals.

Related

• AI providers — vendor credentials for the judge runner and the embedding providers.
• Eval catalog — the question and template library evals draw from.
• Evaluations — the user-facing feature these settings support.