Skip to main content
Monitor’s built-in and custom patterns check traces against rules (does the output contain X, does it match a rubric); online evaluators are different - a real LLM judge scores a sample of your live traffic continuously, the same judge-scoring logic Evaluate’s offline runs use, just pointed at production instead of a golden dataset. This is what most people mean by “online evals”: catching quality drift on real traffic, not just outright failures. Since the judge-scorer unification, an LLM Judge Scorer is ONE entity with three parts: the judge rubric (acceptance/rejection/evaluation criteria, judge prompt, judge model), an offline eval config (how dataset runs grade with it - repetitions, similarity metrics, code scorers), and an online eval config (how it scores live traffic - the “online evaluator” this page describes). Any judge scorer can grade both surfaces. Manage everything from the Scorers page (kind: LLM judge); create one via New scorer → LLM judge scorer, pause its live scoring with the row’s toggle, and open Ratings for a rating-over-time chart. The table sorts newest first.
  • One editor writes all of it atomically, and every save through the editor upserts the online config row - disabled by default, so live scoring is gated by the online config’s Score live traffic switch (off = the config stays saved, nothing is scored live, no LLM spend).
  • On the wire, online: null means the scorer has no online profile yet - an SDK create that omitted online, a seeded template, or a template you just added. A scorer in that state has no live-traffic evidence, so its row menu hides Ratings and shows Tune judge disabled with a tooltip telling you to enable online scoring first; flipping its row toggle opens the editor rather than silently going live - going live is always an explicit save.
  • Engine-seeded quick-start templates (the RAG/Agent/Session/Safety metric packs and the “Example: Helpfulness Judge”) carry seeded: true on the wire; the Scorers page shows its first-run starter state until a non-seeded scorer exists, and lists templates under New scorer → Template scorer (Add clones the template into your own scorer) instead of padding the table.
The editor is a three-step modal:
  1. Judge - the rubric (acceptance / rejection / guidance; acceptance criteria is recommended, not required - offline runs grade against the dataset’s expected results, so the default template alone is a valid judge, while live and session scoring lean on your criteria), with Judge model & prompt as a collapsible machinery card beneath it (model, tool-context level, template - auto-expanded when a custom template exists). The right rail is Try it on real traffic: pick any recent trace or a whole multi-turn session and run ONE judge call with the unsaved draft (GET /agent-monitoring/judge-scorers/preview-context lists the targets, POST .../preview-score scores; sessions go through the sweep’s exact structured session prompt), followed by live assembly cards showing exactly what the judge receives.
  2. Live scoring - the Score live traffic switch, scope (per trace / per session + idle seconds), sample rate, signal threshold and severity, agent restriction, and a judge-calls/day estimate from your last 7 days of traffic.
  3. Offline runs - repetitions, deterministic metrics, code scorers, default-for-runs.
Saving requires only a name. Strictly one online config row per scorer. The legacy SDK/API surfaces (online_evaluators with an evaluation_settings_id reference, and evaluation-settings) keep working as views onto the same entity - the SDK clients now emit a (default-hidden) DeprecationWarning pointing at client.monitor.judge_scorers, and the dashboard no longer uses them anywhere - binding a config that is already another evaluator’s profile transparently binds a fresh copy instead of sharing. A score below the evaluator’s alert threshold (0-10, default 5) raises a signal, the same triage surface a failing Monitor pattern already lands on, deduped by evaluator and agent so a recurring low score accumulates one occurrence count instead of a new signal per trace. Set the threshold’s severity (low/medium/high/critical), or turn the threshold off entirely to score purely for the ratings chart with no triage. A signal raised this way shows the evaluator’s own name and an “LLM judge” tag in the Signals list, distinguishing it from a pattern match at a glance; clicking through opens that evaluator’s ratings dialog directly. The same thing works from the Python SDK:
sample_rate (default 0.1) matters more here than it does for pattern-matching: every check is a real LLM call against your own API key. scope_mode/agent_ids restrict an evaluator to specific agents, same as a pattern’s. Results land in the same event log Overview’s KPI/trend widgets read from, but are excluded from that health-rate math (they’re a continuous score, not a failure signal). For a hard spend ceiling on top of sampling, set AGENTX_QUOTA_ONLINE_JUDGE_CALLS_PER_DAY: a daily per-project cap on all live judge calls - per-trace and per-session evaluators plus scorer-group judge members all draw from the same budget. Unset = unlimited (the broader AGENTX_QUOTA_JUDGE_CALLS_PER_DAY cap - per organization in multi-tenant mode, per instance otherwise - still applies separately).

​
When the judge fails

A judge call that returns nothing usable (empty response, unparseable verdict) is retried once automatically. If the retry also fails, the failure is never disguised as a score:
  • Offline runs: the result is stored with status "skipped" and a null rating - excluded from averageRating and from CI gates, never averaged in as a zero. The run’s liveStatistics carries skippedCount and failedCount so a run that silently judged nothing is visible at a glance.
  • Online scoring: no Signal is raised and no rating is recorded. An online_eval_judge_failure event is kept instead, and the Scorers page shows a per-scorer judge-failure count - a judge that has quietly stopped scoring looks broken there, not healthy.

​
What the judge actually receives

The editable template carries four substitutable variables - {input}, {output}, {expected}, and {context} (retrieval context from metadata.retrievalContext or recorded retrieval spans) - but the judge always sees more than the template. The engine assembles the final message in a fixed order, and a custom prompt can never silently lose the extra context:
  1. Your template, with the four variables substituted.
  2. Agent execution trajectory - tools called with arguments and results, order, failures, rendered from the trace’s span tree (when a trace is linked). Governed by the scorer’s Tool context level (judge.toolContext): "none" strips this block entirely (the judge sees only conversation + expected results - for pure text rubrics), "simple" (the default, the historical behavior) includes it, "detailed" adds the block below.
  3. Tool definitions (toolContext: "detailed" only) - full definitions of each tool the agent actually used, deduped: captured metadata.tools from the trace’s spans first (the exact schema the model saw - the SDK integrations record every request’s tools=[...] list), with the Tools & MCPs registry as a by-name fallback for manual tracer users. Plus one line naming tools advertised to the model but not used, so “was there a better tool it didn’t pick?” stays gradeable without paying for unused schemas. The used-names-only registry lookup is what keeps the project-wide registry safe - it can never inject another agent’s tools.
  4. Judge Guideline - the dataset case’s per-question guideline (offline runs only).
  5. Acceptance Criteria, Rejection Criteria, and Evaluation Criteria (the editor’s “Additional guidance” field) - appended verbatim as labeled blocks.
Two special cases: when a case has no expected results, the default template swaps to a reference-free variant that judges against your criteria alone (a custom template gets a substitution that defuses its expected-results rules in place); and per-session scoring skips the template entirely, building a structured whole-transcript prompt from your criteria - filtered to the same Tool context level ("none" = conversation turns only, "detailed" adds the used-tool definitions to the transcript). The editor’s “What the judge receives” cards render this order live as you edit, with expanders for the exact assembled prompt and a fully filled example.

​
Trajectory-aware judging

Per-trace scoring is trajectory-aware: the judge prompt includes the agent’s actual execution path - tools called with their arguments and results, order, failures - rendered from the trace’s span tree (or its flat tool-call list). Criteria written against the trajectory (“no unnecessary or repeated tool calls”, “answers must follow from tool outputs”) are scoreable, not just aspirational - and the Detailed tool-context level adds the used tools’ full definitions so “was there a better tool it didn’t pick?” is scoreable too.

​
Per-trace vs. per-session scope

By default an evaluator scores individual traces as they arrive (scope="trace"). Pass scope="session" to judge whole conversations instead: a background sweep watches for multi-turn sessions that have been quiet for idle_seconds (default 120), builds the full transcript, and scores it against the same evaluator criteria - so criteria like “did the conversation actually resolve the customer’s problem” become answerable. sample_rate keeps its per-trace meaning here: the keep-or-skip decision is hashed once per (conversation, scorer) and is stable across sweep ticks, so a sampled-out conversation stays out permanently and the configured rate is the true fraction of conversations judged - not a per-tick roll that would eventually score everything. See Multi-Turn Session Evaluation for the full guide: sweep mechanics, the built-in Session Baseline Judge, on-demand SDK checks, and turning failing conversations into regression tests. Every project ships with one built-in session evaluator: Session Baseline Judge, created paused like every other scorer - switch it on from the Scorers page. It scores whole-conversation consistency - context retention, self-contradiction, goal drift - and points at the first step where the conversation broke. Its rubric is a normal judge rubric (not code), so Tune judge improves it from calibration evidence like any other scorer; the scorer itself is read-only except the rubric, the prompt, and the live-scoring switch, and can’t be deleted.
The dashboard’s evaluator editor has the same Scope card (Per trace / Per session). AGENTX_SESSION_SWEEP=false disables the background sweep; POST /agent-monitoring/session-sweep/run triggers one manually either way. Full lifecycle from the SDK - list/get/update/delete alongside the builder, plus the scored evidence behind each evaluator:

​
Tuning the judge from the SDK

The dashboard’s Tune judge flow (measure the evaluator against recorded reality, rewrite its criteria from the disagreements, validate by exact re-judging, publish behind approval) is fully scriptable:
The window argument accepts "24h", "7d", "30d", or "rubric". "rubric" - the dashboard’s Tune Judge default - means verdicts produced by the current rubric: everything since the scorer’s criteria were last edited or a tuning proposal was published, clamped to 30 days. That is the semantically right evidence set for tuning, because disagreements older than the last rewrite are complaints about criteria that no longer exist, and tuning from them re-litigates issues a previous rewrite already fixed. The time windows remain useful for scorers with sparse review coverage, where the current rubric hasn’t accumulated ground truth yet. The response’s window and since fields echo the boundary that was actually applied.

​
Reading the agreement numbers

agreementRate is raw agreement, and raw agreement flatters a judge on imbalanced traffic: if 90% of responses are genuinely fine, a judge that passes everything scores 90% while detecting nothing. alpha is the same comparison corrected for chance - Krippendorff’s alpha over the binary verdict pair, which for two raters is Scott’s pi with a small-sample correction. Read it as: 1 = perfect alignment, 0 = no better than a coin weighted like the traffic, negative = systematically opposed. alphaBand ships a plain-language band (poor / slight / fair / moderate / substantial / near-perfect) so every surface interprets the same value the same way. Two honesty rules. First, alpha is null until at least 15 compared items have ground truth (alphaMinItems echoes the threshold) - with a handful of labels the statistic swings wildly, so it is withheld rather than fabricated. Second, the ground-truth streams (triage corrections, review labels, disputes) skew toward contested cases by construction, so read alpha as alignment on human-reviewed items, not on raw traffic - a genuinely fine judge can look mediocre if humans only ever reviewed the hard calls. Sampled labels from the review queue reduce that skew. Ground truth comes from outcomes, user feedback, and human review-queue labels; the validation verdict is measured (candidate criteria re-judge the exact cases the current ones got wrong, plus a control set they got right), not estimated. Publishing is gated on that provenance: pass the validate_tuning result as validation, and a "regressed" verdict is refused - force=True is the deliberate escape hatch, recorded as such in the version history. The provenance gate is enforced with a signed token. validate_tuning returns a validationToken proving the verdict was measured for exactly that candidate package on that scorer, and publish requires it - the SDK passes it automatically when you hand the validate_tuning result to publish_tuning. The token expires after 24 hours; a publish after that returns 409, and you re-run validate_tuning on the same package to get a fresh one.

​
Built-in metric pack

Every project ships with seeded judge-scorer templates (seeded: true on the wire) - twelve across four families (RAG, Agent, Session, Safety; the full list is on the Scorers page). They are pickable as the grader anywhere one is chosen (dataset runs, the Playground) and live in New scorer → Template scorer rather than padding the Scorers table - Add clones one into your own editable, deletable scorer. The RAG five:
  • RAG: Faithfulness - every factual claim in the response must be supported by the retrieved context.
  • RAG: Answer Relevancy - does the response actually address the query?
  • RAG: Context Relevancy - judges the retriever: were the retrieved chunks relevant and sufficient? A low score means fix retrieval, not the prompt.
  • RAG: Contextual Precision - are the relevant chunks ranked above the irrelevant ones? Targets the reranker.
  • RAG: Contextual Recall - does the context cover what the expected answer needs? Offline-oriented (needs a case’s expected_results).
The RAG prompts reference {context}, which resolves from a trace’s metadata.retrievalContext (a string or an array of chunk strings) when set, else from the trace’s own recorded retrieval spans (tracer.trace_retrieval(...), LangChain/LlamaIndex retriever callbacks, OTel retrieval spans) - so if your integration already records retrievals, the judges see the chunks with no caller changes. See RAG evaluation for the full metric guide and the offline context sources. Add one from the template picker to get an ordinary scorer you can edit, tune with Tune judge, or delete - the template itself stays in the library. A zero-cost PII in response built-in pattern (emails, phone numbers, SSNs, payment cards, regex-based) also runs alongside the other built-in checks.