Skip to main content
Monitor’s built-in and custom patterns check traces against rules (does the output contain X, does it match a rubric); online evaluators are different - a real LLM judge scores a sample of your live traffic continuously, the same judge-scoring logic Evaluate’s offline runs use, just pointed at production instead of a golden dataset. This is what most people mean by “online evals”: catching quality drift on real traffic, not just outright failures. Manage them from the Scorers page (kind: LLM judge; formerly called “online evaluators”, which the SDK and API still use as the resource name). Create one via New scorer → LLM judge scorer with a name and a reference to an existing Evaluator config (its criteria, judge prompt, and judge model, the same config datasets/Evaluate runs use, editable from Evaluate → Evaluator), a sample rate, and an optional agent scope; pause it with the row’s toggle, edit or delete it from the row menu, and open Ratings for a rating-over-time chart. A score below the evaluator’s alert threshold (0-10, default 5) raises a signal, the same triage surface a failing Monitor pattern already lands on, deduped by evaluator and agent so a recurring low score accumulates one occurrence count instead of a new signal per trace. Set the threshold’s severity (low/medium/high/critical), or turn the threshold off entirely to score purely for the ratings chart with no triage. A signal raised this way shows the evaluator’s own name and an “LLM judge” tag in the Signals list, distinguishing it from a pattern match at a glance; clicking through opens that evaluator’s ratings dialog directly. The same thing works from the Python SDK:
sample_rate (default 0.1) matters more here than it does for pattern-matching: every check is a real LLM call against your own API key. scope_mode/agent_ids restrict an evaluator to specific agents, same as a pattern’s. Results land in the same event log Overview’s KPI/trend widgets read from, but are excluded from that health-rate math (they’re a continuous score, not a failure signal).

​
Trajectory-aware judging

Per-trace scoring is trajectory-aware: the judge prompt includes the agent’s actual execution path - tools called with their arguments and results, order, failures - rendered from the trace’s span tree (or its flat tool-call list). Criteria written against the trajectory (“no unnecessary or repeated tool calls”, “answers must follow from tool outputs”) are scoreable, not just aspirational; the evaluator config form ships a one-click trajectory accuracy criteria template for exactly this.

​
Per-trace vs. per-session scope

By default an evaluator scores individual traces as they arrive (scope="trace"). Pass scope="session" to judge whole conversations instead: a background sweep watches for multi-turn sessions that have been quiet for idle_seconds (default 120), builds the full transcript, and scores it against the same evaluator criteria - so criteria like “did the conversation actually resolve the customer’s problem” become answerable. See Multi-Turn Session Evaluation for the full guide: sweep mechanics, the built-in Session Baseline Judge, on-demand SDK checks, and turning failing conversations into regression tests. Every project ships with one built-in session evaluator: Session Baseline Judge. It scores whole-conversation consistency - context retention, self-contradiction, goal drift - and points at the first step where the conversation broke. Its rubric is a normal evaluator config (not code), so Tune judge improves it from calibration evidence like any other evaluator; the evaluator row itself is read-only except pause/enable, and can’t be deleted.
The dashboard’s evaluator editor has the same Scope card (Per trace / Per session). AGENTX_SESSION_SWEEP=false disables the background sweep; POST /agent-monitoring/session-sweep/run triggers one manually either way. Full lifecycle from the SDK - list/get/update/delete alongside the builder, plus the scored evidence behind each evaluator:

​
Tuning the judge from the SDK

The dashboard’s Tune judge flow (measure the evaluator against recorded reality, rewrite its criteria from the disagreements, validate by exact re-judging, publish behind approval) is fully scriptable:
Ground truth comes from outcomes and user feedback; the validation verdict is measured (candidate criteria re-judge the exact cases the current ones got wrong, plus a control set they got right), not estimated.

​
Built-in metric pack

Every project ships with ready-made evaluator configs, usable anywhere a config is picked (online evaluators, dataset runs, the Playground):
  • RAG: Faithfulness - every factual claim in the response must be supported by the retrieved context.
  • RAG: Answer Relevancy - does the response actually address the query?
  • RAG: Context Relevancy - judges the retriever: were the retrieved chunks relevant and sufficient? A low score means fix retrieval, not the prompt.
  • RAG: Contextual Precision - are the relevant chunks ranked above the irrelevant ones? Targets the reranker.
  • RAG: Contextual Recall - does the context cover what the expected answer needs? Offline-oriented (needs a case’s expected_results).
The RAG prompts reference {context}, which resolves from a trace’s metadata.retrievalContext (a string or an array of chunk strings) when set, else from the trace’s own recorded retrieval spans (tracer.trace_retrieval(...), LangChain/LlamaIndex retriever callbacks, OTel retrieval spans) - so if your integration already records retrievals, the judges see the chunks with no caller changes. See RAG evaluation for the full metric guide and the offline context sources. They’re ordinary configs: edit them, tune them with Tune judge, or delete them - a deletion stays deleted. A zero-cost PII in response built-in pattern (emails, phone numbers, SSNs, payment cards, regex-based) also runs alongside the other built-in checks.