> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Judge Scorers

> A real LLM judge scoring live traffic continuously, per trace or per session

Monitor's built-in and custom patterns check traces against rules (does the output contain X, does it match a rubric); online evaluators are different - a real LLM judge scores a sample of your live traffic continuously, the same judge-scoring logic Evaluate's offline runs use, just pointed at production instead of a golden dataset. This is what most people mean by "online evals": catching quality drift on real traffic, not just outright failures.

Manage them from the [Scorers page](/monitor/scorers) (kind: **LLM judge**; formerly called "online evaluators", which the SDK and API still use as the resource name). Create one via **New scorer** → **LLM judge scorer** with a name and a reference to an existing Evaluator config (its criteria, judge prompt, and judge model, the same config datasets/Evaluate runs use, editable from Evaluate → **Evaluator**), a sample rate, and an optional agent scope; pause it with the row's toggle, edit or delete it from the row menu, and open **Ratings** for a rating-over-time chart.

A score below the evaluator's **alert threshold** (0-10, default 5) raises a signal, the same triage surface a failing Monitor pattern already lands on, deduped by evaluator and agent so a recurring low score accumulates one occurrence count instead of a new signal per trace. Set the threshold's **severity** (low/medium/high/critical), or turn the threshold off entirely to score purely for the ratings chart with no triage. A signal raised this way shows the evaluator's own name and an "LLM judge" tag in the Signals list, distinguishing it from a pattern match at a glance; clicking through opens that evaluator's ratings dialog directly.

The same thing works from the Python SDK:

```python theme={null}
evaluator = client.monitor.online_evaluators.builder(
    name="Helpfulness",
    evaluation_settings_id=settings.id,
    sample_rate=0.1,
    alert_threshold=5,
    severity="medium",
).publish()

client.monitor.online_evaluators.get(evaluator.id)
client.monitor.online_evaluators.list()
client.monitor.online_evaluators.update(evaluator.id, alert_threshold=None)  # score only, never raise a signal
client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
client.monitor.online_evaluators.events(evaluator.id, window="7d")  # individually scored traces behind a ratings point
client.monitor.online_evaluators.delete(evaluator.id)
```

`sample_rate` (default `0.1`) matters more here than it does for pattern-matching: every check is a real LLM call against your own API key. `scope_mode`/`agent_ids` restrict an evaluator to specific agents, same as a pattern's. Results land in the same event log Overview's KPI/trend widgets read from, but are excluded from that health-rate math (they're a continuous score, not a failure signal).

### Trajectory-aware judging

Per-trace scoring is trajectory-aware: the judge prompt includes the agent's actual execution
path - tools called with their arguments and results, order, failures - rendered from the
trace's span tree (or its flat tool-call list). Criteria written against the trajectory
("no unnecessary or repeated tool calls", "answers must follow from tool outputs") are
scoreable, not just aspirational; the evaluator config form ships a one-click **trajectory
accuracy** criteria template for exactly this.

### Per-trace vs. per-session scope

By default an evaluator scores individual traces as they arrive (`scope="trace"`). Pass `scope="session"` to judge **whole conversations** instead: a background sweep watches for multi-turn sessions that have been quiet for `idle_seconds` (default 120), builds the full transcript, and scores it against the same evaluator criteria - so criteria like "did the conversation actually resolve the customer's problem" become answerable. See [Multi-Turn Session Evaluation](/monitor/session-evaluation) for the full guide: sweep mechanics, the built-in Session Baseline Judge, on-demand SDK checks, and turning failing conversations into regression tests.

Every project ships with one built-in session evaluator: **Session Baseline Judge**. It scores whole-conversation consistency - context retention, self-contradiction, goal drift - and points at the first step where the conversation broke. Its rubric is a normal evaluator config (not code), so **Tune judge** improves it from calibration evidence like any other evaluator; the evaluator row itself is read-only except pause/enable, and can't be deleted.

```python theme={null}
client.monitor.online_evaluators.builder(
    name="Conversation resolution",
    evaluation_settings_id=settings.id,
    scope="session",
    idle_seconds=120,
    alert_threshold=5,
).publish()
```

The dashboard's evaluator editor has the same Scope card (Per trace / Per session). `AGENTX_SESSION_SWEEP=false` disables the background sweep; `POST /agent-monitoring/session-sweep/run` triggers one manually either way.

Full lifecycle from the SDK - `list`/`get`/`update`/`delete` alongside the builder, plus the
scored evidence behind each evaluator:

```python theme={null}
evaluators = client.monitor.online_evaluators.list()
client.monitor.online_evaluators.update(evaluator_id, sample_rate=0.25)  # or enabled=False to pause

ratings = client.monitor.online_evaluators.ratings(evaluator_id, window="7d")  # bucketed averages
events = client.monitor.online_evaluators.events(evaluator_id, window="7d")    # worst-rated traces + justifications
```

## Tuning the judge from the SDK

The dashboard's **Tune judge** flow (measure the evaluator against recorded reality, rewrite
its criteria from the disagreements, validate by exact re-judging, publish behind approval) is
fully scriptable:

```python theme={null}
cal = client.monitor.online_evaluators.calibration(evaluator_id, window="7d")
# cal["agreements"], cal["missed"], cal["overFlagged"], cal["disagreementCases"]

proposal = client.monitor.online_evaluators.tune(evaluator_id, window="7d")
criteria = {k: proposal[k] for k in ("acceptanceCriteria", "rejectionCriteria", "evaluationCriteria")}

verdict = client.monitor.online_evaluators.validate_tuning(evaluator_id, criteria, window="7d")
if verdict["verdict"] == "improved":
    client.monitor.online_evaluators.publish_tuning(evaluator_id, criteria)
```

Ground truth comes from [outcomes and user feedback](/monitor/outcomes); the validation verdict
is measured (candidate criteria re-judge the exact cases the current ones got wrong, plus a
control set they got right), not estimated.

## Built-in metric pack

Every project ships with ready-made evaluator configs, usable anywhere a config is picked (online evaluators, dataset runs, the Playground):

* **RAG: Faithfulness** - every factual claim in the response must be supported by the retrieved context.
* **RAG: Answer Relevancy** - does the response actually address the query?
* **RAG: Context Relevancy** - judges the *retriever*: were the retrieved chunks relevant and sufficient? A low score means fix retrieval, not the prompt.
* **RAG: Contextual Precision** - are the relevant chunks ranked above the irrelevant ones? Targets the reranker.
* **RAG: Contextual Recall** - does the context cover what the expected answer needs? Offline-oriented (needs a case's `expected_results`).

The RAG prompts reference `{context}`, which resolves from a trace's `metadata.retrievalContext` (a string or an array of chunk strings) when set, else from the trace's own recorded **retrieval spans** (`tracer.trace_retrieval(...)`, LangChain/LlamaIndex retriever callbacks, OTel retrieval spans) - so if your integration already records retrievals, the judges see the chunks with no caller changes. See [RAG evaluation](/evaluation/rag) for the full metric guide and the offline context sources. They're ordinary configs: edit them, tune them with **Tune judge**, or delete them - a deletion stays deleted. A zero-cost **PII in response** built-in pattern (emails, phone numbers, SSNs, payment cards, regex-based) also runs alongside the other built-in checks.