> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Judge Scorers

> A real LLM judge scoring live traffic continuously, per trace or per session

Monitor's built-in and custom patterns check traces against rules (does the output contain X, does it match a rubric); online evaluators are different - a real LLM judge scores a sample of your live traffic continuously, the same judge-scoring logic Evaluate's offline runs use, just pointed at production instead of a golden dataset. This is what most people mean by "online evals": catching quality drift on real traffic, not just outright failures.

Since the judge-scorer unification, an LLM Judge Scorer is ONE entity with three parts: the **judge rubric** (acceptance/rejection/evaluation criteria, judge prompt, judge model), an **offline eval config** (how dataset runs grade with it - repetitions, similarity metrics, code scorers), and an **online eval config** (how it scores live traffic - the "online evaluator" this page describes). Any judge scorer can grade both surfaces. Manage everything from the [Scorers page](/monitor/scorers) (kind: **LLM judge**); create one via **New scorer** → **LLM judge scorer**, pause its live scoring with the row's toggle, and open **Ratings** for a rating-over-time chart. The table sorts newest first.

* **One editor writes all of it atomically**, and every save through the editor upserts the online config row - disabled by default, so live scoring is gated by the online config's **Score live traffic** switch (off = the config stays saved, nothing is scored live, no LLM spend).
* **On the wire, `online: null` means the scorer has no online profile yet** - an SDK create that omitted `online`, a seeded template, or a template you just added. A scorer in that state has no live-traffic evidence, so its row menu hides **Ratings** and shows **Tune judge** disabled with a tooltip telling you to enable online scoring first; flipping its row toggle opens the editor rather than silently going live - going live is always an explicit save.
* **Engine-seeded quick-start templates** (the RAG/Agent/Session/Safety metric packs and the "Example: Helpfulness Judge") carry `seeded: true` on the wire; the Scorers page shows its first-run starter state until a non-seeded scorer exists, and lists templates under **New scorer → Template scorer** (Add clones the template into your own scorer) instead of padding the table.

The editor is a three-step modal:

1. **Judge** - the rubric (acceptance / rejection / guidance; acceptance criteria is
   *recommended, not required* - offline runs grade against the dataset's expected results, so
   the default template alone is a valid judge, while live and session scoring lean on your
   criteria), with **Judge model & prompt** as a collapsible machinery card beneath it (model,
   tool-context level, template - auto-expanded when a custom template exists). The right rail
   is **Try it on real traffic**: pick any recent trace *or a whole multi-turn session* and run
   ONE judge call with the unsaved draft (`GET /agent-monitoring/judge-scorers/preview-context`
   lists the targets, `POST .../preview-score` scores; sessions go through the sweep's exact
   structured session prompt), followed by live assembly cards showing exactly what the judge
   receives.
2. **Live scoring** - the Score live traffic switch, scope (per trace / per session + idle
   seconds), sample rate, signal threshold and severity, agent restriction, and a
   judge-calls/day estimate from your last 7 days of traffic.
3. **Offline runs** - repetitions, deterministic metrics, code scorers, default-for-runs.

Saving requires only a name.

Strictly one online config row per scorer. The legacy SDK/API surfaces (`online_evaluators` with an `evaluation_settings_id` reference, and `evaluation-settings`) keep working as views onto the same entity - the SDK clients now emit a (default-hidden) DeprecationWarning pointing at `client.monitor.judge_scorers`, and the dashboard no longer uses them anywhere - binding a config that is already another evaluator's profile transparently binds a fresh copy instead of sharing.

A score below the evaluator's **alert threshold** (0-10, default 5) raises a signal, the same triage surface a failing Monitor pattern already lands on, deduped by evaluator and agent so a recurring low score accumulates one occurrence count instead of a new signal per trace. Set the threshold's **severity** (low/medium/high/critical), or turn the threshold off entirely to score purely for the ratings chart with no triage. A signal raised this way shows the evaluator's own name and an "LLM judge" tag in the Signals list, distinguishing it from a pattern match at a glance; clicking through opens that evaluator's ratings dialog directly.

The same thing works from the Python SDK:

```python theme={null}
# The unified surface (preferred): rubric + profiles in one call.
scorer = client.monitor.judge_scorers.create(
    "Helpfulness",
    judge={"acceptanceCriteria": "Actually resolves the user's question."},
    online={"enabled": True, "sampleRate": 0.1, "alertThreshold": 5, "severity": "medium"},
)
client.monitor.judge_scorers.ratings(scorer.id, window="7d")
client.evaluations.run(dataset_id, subject, scorer_id=scorer.id)  # same rubric offline

# Or the snake_case builder - rubric, offline config, and live profile in one call:
scorer = client.monitor.judge_scorers.builder(
    "Helpfulness", acceptance_criteria="Actually resolves the user's question.",
    live=True, sample_rate=0.1, alert_threshold=5,
).publish()

# The legacy per-profile surface still works unchanged:
evaluator = client.monitor.online_evaluators.builder(
    name="Helpfulness",
    evaluation_settings_id=settings.id,
    sample_rate=0.1,
    alert_threshold=5,
    severity="medium",
).publish()

client.monitor.online_evaluators.get(evaluator.id)
client.monitor.online_evaluators.list()
client.monitor.online_evaluators.update(evaluator.id, alert_threshold=None)  # score only, never raise a signal
client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
client.monitor.online_evaluators.events(evaluator.id, window="7d")  # individually scored traces behind a ratings point
client.monitor.online_evaluators.delete(evaluator.id)
```

`sample_rate` (default `0.1`) matters more here than it does for pattern-matching: every check is a real LLM call against your own API key. `scope_mode`/`agent_ids` restrict an evaluator to specific agents, same as a pattern's. Results land in the same event log Overview's KPI/trend widgets read from, but are excluded from that health-rate math (they're a continuous score, not a failure signal). For a hard spend ceiling on top of sampling, set `AGENTX_QUOTA_ONLINE_JUDGE_CALLS_PER_DAY`: a daily per-project cap on all live judge calls - per-trace and per-session evaluators plus [scorer-group](/monitor/scorer-groups) judge members all draw from the same budget. Unset = unlimited (the broader `AGENTX_QUOTA_JUDGE_CALLS_PER_DAY` cap - per organization in multi-tenant mode, per instance otherwise - still applies separately).

### When the judge fails

A judge call that returns nothing usable (empty response, unparseable verdict) is retried once
automatically. If the retry also fails, the failure is never disguised as a score:

* **Offline runs**: the result is stored with status `"skipped"` and a `null` rating - excluded
  from `averageRating` and from [CI gates](/integrations/self-host-ci), never averaged in as a
  zero. The run's `liveStatistics` carries `skippedCount` and `failedCount` so a run that
  silently judged nothing is visible at a glance.
* **Online scoring**: no Signal is raised and no rating is recorded. An
  `online_eval_judge_failure` event is kept instead, and the
  [Scorers page](/monitor/scorers) shows a per-scorer judge-failure count - a judge that has
  quietly stopped scoring looks broken there, not healthy.

### What the judge actually receives

The editable template carries four substitutable variables - `{input}`, `{output}`,
`{expected}`, and `{context}` (retrieval context from `metadata.retrievalContext` or recorded
retrieval spans) - but the judge always sees more than the template. The engine assembles the
final message in a fixed order, and a custom prompt can never silently lose the extra context:

1. **Your template**, with the four variables substituted.
2. **Agent execution trajectory** - tools called with arguments and results, order, failures,
   rendered from the trace's span tree (when a trace is linked). Governed by the scorer's
   **Tool context** level (`judge.toolContext`): `"none"` strips this block entirely (the
   judge sees only conversation + expected results - for pure text rubrics), `"simple"` (the
   default, the historical behavior) includes it, `"detailed"` adds the block below.
3. **Tool definitions** (`toolContext: "detailed"` only) - full definitions of each tool the
   agent actually *used*, deduped: captured `metadata.tools` from the trace's spans first
   (the exact schema the model saw - the SDK integrations record every request's `tools=[...]`
   list), with the Tools & MCPs registry as a by-name fallback for manual tracer users. Plus
   one line naming tools advertised to the model but not used, so "was there a better tool it
   didn't pick?" stays gradeable without paying for unused schemas. The used-names-only
   registry lookup is what keeps the project-wide registry safe - it can never inject another
   agent's tools.
4. **Judge Guideline** - the dataset case's per-question guideline (offline runs only).
5. **Acceptance Criteria**, **Rejection Criteria**, and **Evaluation Criteria** (the editor's
   "Additional guidance" field) - appended verbatim as labeled blocks.

Two special cases: when a case has no expected results, the default template swaps to a
reference-free variant that judges against your criteria alone (a custom template gets a
substitution that defuses its expected-results rules in place); and per-session scoring skips
the template entirely, building a structured whole-transcript prompt from your criteria -
filtered to the same Tool context level (`"none"` = conversation turns only, `"detailed"` adds
the used-tool definitions to the transcript). The
editor's "What the judge receives" cards render this order live as you edit, with expanders for
the exact assembled prompt and a fully filled example.

### Trajectory-aware judging

Per-trace scoring is trajectory-aware: the judge prompt includes the agent's actual execution
path - tools called with their arguments and results, order, failures - rendered from the
trace's span tree (or its flat tool-call list). Criteria written against the trajectory
("no unnecessary or repeated tool calls", "answers must follow from tool outputs") are
scoreable, not just aspirational - and the **Detailed** tool-context level adds the used tools'
full definitions so "was there a better tool it didn't pick?" is scoreable too.

### Per-trace vs. per-session scope

By default an evaluator scores individual traces as they arrive (`scope="trace"`). Pass `scope="session"` to judge **whole conversations** instead: a background sweep watches for multi-turn sessions that have been quiet for `idle_seconds` (default 120), builds the full transcript, and scores it against the same evaluator criteria - so criteria like "did the conversation actually resolve the customer's problem" become answerable. `sample_rate` keeps its per-trace meaning here: the keep-or-skip decision is hashed once per (conversation, scorer) and is stable across sweep ticks, so a sampled-out conversation stays out permanently and the configured rate is the true fraction of conversations judged - not a per-tick roll that would eventually score everything. See [Multi-Turn Session Evaluation](/monitor/session-evaluation) for the full guide: sweep mechanics, the built-in Session Baseline Judge, on-demand SDK checks, and turning failing conversations into regression tests.

Every project ships with one built-in session evaluator: **Session Baseline Judge**, created paused like every other scorer - switch it on from the Scorers page. It scores whole-conversation consistency - context retention, self-contradiction, goal drift - and points at the first step where the conversation broke. Its rubric is a normal judge rubric (not code), so **Tune judge** improves it from calibration evidence like any other scorer; the scorer itself is read-only except the rubric, the prompt, and the live-scoring switch, and can't be deleted.

```python theme={null}
client.monitor.judge_scorers.builder(
    "Conversation resolution",
    acceptance_criteria="The customer's problem is actually resolved by the end.",
    live=True,
    scope="session",
    idle_seconds=120,
    sample_rate=0.1,   # true fraction of conversations judged - deterministic per conversation
    alert_threshold=5,
).publish()
```

The dashboard's evaluator editor has the same Scope card (Per trace / Per session). `AGENTX_SESSION_SWEEP=false` disables the background sweep; `POST /agent-monitoring/session-sweep/run` triggers one manually either way.

Full lifecycle from the SDK - `list`/`get`/`update`/`delete` alongside the builder, plus the
scored evidence behind each evaluator:

```python theme={null}
evaluators = client.monitor.online_evaluators.list()
client.monitor.online_evaluators.update(evaluator_id, sample_rate=0.25)  # or enabled=False to pause

ratings = client.monitor.online_evaluators.ratings(evaluator_id, window="7d")  # bucketed averages
events = client.monitor.online_evaluators.events(evaluator_id, window="7d")    # worst-rated traces + justifications
```

## Tuning the judge from the SDK

The dashboard's **Tune judge** flow (measure the evaluator against recorded reality, rewrite
its criteria from the disagreements, validate by exact re-judging, publish behind approval) is
fully scriptable:

```python theme={null}
cal = client.monitor.online_evaluators.calibration(evaluator_id, window="rubric")
# cal["agreements"], cal["missed"], cal["overFlagged"], cal["disagreementCases"]
# cal["alpha"] - chance-corrected agreement (see below); cal["ratingMae"] - mean absolute
# error against human re-scores, over the pairs that carry a corrected number

proposal = client.monitor.online_evaluators.tune(evaluator_id, window="7d")
criteria = {k: proposal[k] for k in ("acceptanceCriteria", "rejectionCriteria", "evaluationCriteria")}

verdict = client.monitor.online_evaluators.validate_tuning(evaluator_id, criteria, window="7d")
client.monitor.online_evaluators.publish_tuning(evaluator_id, criteria, validation=verdict)
```

The `window` argument accepts `"24h"`, `"7d"`, `"30d"`, or `"rubric"`. `"rubric"` - the
dashboard's Tune Judge default - means *verdicts produced by the current rubric*: everything
since the scorer's criteria were last edited or a tuning proposal was published, clamped to 30
days. That is the semantically right evidence set for tuning, because disagreements older than
the last rewrite are complaints about criteria that no longer exist, and tuning from them
re-litigates issues a previous rewrite already fixed. The time windows remain useful for
scorers with sparse review coverage, where the current rubric hasn't accumulated ground truth
yet. The response's `window` and `since` fields echo the boundary that was actually applied.

### Reading the agreement numbers

`agreementRate` is raw agreement, and raw agreement flatters a judge on imbalanced traffic: if
90% of responses are genuinely fine, a judge that passes everything scores 90% while detecting
nothing. `alpha` is the same comparison corrected for chance - Krippendorff's alpha over the
binary verdict pair, which for two raters is Scott's pi with a small-sample correction. Read it
as: 1 = perfect alignment, 0 = no better than a coin weighted like the traffic, negative =
systematically opposed. `alphaBand` ships a plain-language band (poor / slight / fair /
moderate / substantial / near-perfect) so every surface interprets the same value the same way.

Two honesty rules. First, `alpha` is `null` until at least 15 compared items have ground truth
(`alphaMinItems` echoes the threshold) - with a handful of labels the statistic swings wildly,
so it is withheld rather than fabricated.
Second, the ground-truth streams (triage corrections, review labels, disputes) skew toward
contested cases by construction, so read `alpha` as *alignment on human-reviewed items*, not on
raw traffic - a genuinely fine judge can look mediocre if humans only ever reviewed the hard
calls. Sampled labels from the [review queue](/monitor/review-queue) reduce that skew.

Ground truth comes from [outcomes, user feedback](/monitor/outcomes), and
[human review-queue labels](/monitor/review-queue); the validation verdict
is measured (candidate criteria re-judge the exact cases the current ones got wrong, plus a
control set they got right), not estimated. Publishing is gated on that provenance: pass the
`validate_tuning` result as `validation`, and a `"regressed"` verdict is refused - `force=True`
is the deliberate escape hatch, recorded as such in the version history.

The provenance gate is enforced with a signed token. `validate_tuning` returns a
`validationToken` proving the verdict was measured for exactly that candidate package on that
scorer, and publish requires it - the SDK passes it automatically when you hand the
`validate_tuning` result to `publish_tuning`. The token expires after 24 hours; a publish
after that returns 409, and you re-run `validate_tuning` on the same package to get a fresh
one.

## Built-in metric pack

Every project ships with seeded judge-scorer templates (`seeded: true` on the wire) - twelve across four families (RAG, Agent, Session, Safety; the full list is on the [Scorers page](/monitor/scorers)). They are pickable as the grader anywhere one is chosen (dataset runs, the Playground) and live in **New scorer → Template scorer** rather than padding the Scorers table - **Add** clones one into your own editable, deletable scorer. The RAG five:

* **RAG: Faithfulness** - every factual claim in the response must be supported by the retrieved context.
* **RAG: Answer Relevancy** - does the response actually address the query?
* **RAG: Context Relevancy** - judges the *retriever*: were the retrieved chunks relevant and sufficient? A low score means fix retrieval, not the prompt.
* **RAG: Contextual Precision** - are the relevant chunks ranked above the irrelevant ones? Targets the reranker.
* **RAG: Contextual Recall** - does the context cover what the expected answer needs? Offline-oriented (needs a case's `expected_results`).

The RAG prompts reference `{context}`, which resolves from a trace's `metadata.retrievalContext` (a string or an array of chunk strings) when set, else from the trace's own recorded **retrieval spans** (`tracer.trace_retrieval(...)`, LangChain/LlamaIndex retriever callbacks, OTel retrieval spans) - so if your integration already records retrievals, the judges see the chunks with no caller changes. See [RAG evaluation](/evaluation/rag) for the full metric guide and the offline context sources. Add one from the template picker to get an ordinary scorer you can edit, tune with **Tune judge**, or delete - the template itself stays in the library. A zero-cost **PII in response** built-in pattern (emails, phone numbers, SSNs, payment cards, regex-based) also runs alongside the other built-in checks.