Skip to main content
A scorer is anything that can pass judgment on your agent’s traffic - and the same scorer serves both evaluation surfaces: it can grade live traffic (online evaluation) and dataset runs (offline evaluation). That is why Scorers is its own docs category rather than a sub-page of either one. The Scorers page is the single catalog for all of them - one list, filtered by intent rather than split by implementation, with an inline enable toggle per row. Three kinds live side by side: Everything is opt-in: a fresh project applies no scorer at all, and the page’s footer says so plainly - nothing is judged until a scorer is enabled. Operational outcomes (trace errors, tool failures, empty responses) are not scorers and need no enabling: the trace itself recorded them, and they feed the KPI failure metrics directly.

​
In this section

The seeded template catalog (New scorer -> Template scorer) ships four families: RAG (Faithfulness, Answer Relevancy, Context Relevancy, Contextual Precision, Contextual Recall), Agent (Task Completion, Tool Correctness, Step Efficiency - trajectory-anchored), Session (Knowledge Retention, Role Adherence - enable with per-session scope), and Safety (Harmful Content, Bias & Fairness - meaning-level judges next to the zero-cost content patterns).

​
The two grading modes

The same scorer runs on two surfaces with one structural difference: every offline dataset case carries expected_results; no live trace ever does. The judge handles this with one rubric and two modes: Because both modes anchor on the criteria, a 7 online and a 7 offline answer the same question (“does this meet the criteria?”) - the reference just adds factual strictness where one exists. This is the industry “reference-guided grading” pattern, applied so one scorer honestly serves both surfaces instead of you maintaining two. The exception: a rubric that is about the reference itself - RAG: Contextual Recall (“does the context cover what the expected answer needs?”) or a custom “matches the reference exactly” prompt - has nothing to grade without one. Mark it Needs a reference answer in the editor: the engine then refuses to enable live scoring for it (HTTP 409) and offline runs skip any case that lacks expected_results with an explicit reason, instead of judging emptiness.

​
The list

  • Stat strip: how many scorers are enabled, how many are firing this week, total signal occurrences in the window, and how many are available but off.
  • Filters: scope pills (All / Enabled / Firing / Off), a kind filter, and search. Rows sort newest first.
  • The table is yours: it lists what you created or enabled. Pre-existing material - built-in patterns, seeded judge templates, the built-in session judge - stays in the Template scorer picker until enabled or added, so a new project’s table holds exactly what you made.
  • Signals · 7d: the payoff column - each enabled scorer’s signal count for the week with a daily sparkline, “None yet” for enabled-but-quiet, “Not running” for disabled.
  • Enabled: a toggle, right on the row. Built-in templates store this project-wide (enabledBuiltinPatterns in monitoring defaults); the other kinds carry their own flag.
  • Row actions: click a row (or its menu) to open the scorer’s editor - built-in templates open read-only with an “Exactly what it checks” list of their literal rules; LLM judges also get Ratings (score-over-time) and Tune judge (rewrite criteria from calibration evidence). Both are live-traffic surfaces, so a judge with no online profile yet hides Ratings and shows Tune judge disabled, with a tooltip telling you to enable online scoring and let it score some traffic first.
New scorer offers the four authorable kinds - a pattern (deterministic rules), an LLM judge (criteria in plain language + a judge model), a code scorer (your own Python/JavaScript handler(), run in-engine), or an external endpoint - plus Template scorer: the library of ready-made material. Adding an LLM judge template clones it into your own editable copy; built-in patterns and the session judge switch on in place.

​
First run

For a clean project - nothing enabled and nothing of your own created yet - the page opens as a starting point instead of a wall of paused rows: recommended templates to switch on with one click (all zero-LLM-cost, plus the built-in Session Baseline Judge), or jump straight into authoring your own. Creating any scorer of your own exits this state immediately, even with live scoring off.

​
Where scorer verdicts land

An enabled scorer’s hits raise signals - deduped rows queued for verdicts on the Review tab, with Monitor charting their volume and each live judge’s score trend. Judge scorers additionally record a rating per sampled trace whether or not it crossed the alert threshold, feeding Judge Calibration against outcomes and user feedback.

​
What moves which metric

Not everything that raises a signal reclassifies the run - build alerting on the right surface: The practical consequence: an alert built on failureRate will see operational failures and pattern hits, but NOT code-scorer, external-scorer, or judge verdicts - watch those scorers’ own signals and per-scorer event history (or the Scorers page’s Signals column) instead.

​
Scorers score, rules route

A neighboring concept worth keeping straight: an automation rule also watches live traffic and also has a sample rate, but it never scores anything. It routes matching traces into human review, into a dataset, or out to a webhook. Scoring costs a judge call and belongs to a scorer; routing is free and belongs to a rule. Enabling a rule therefore cannot change what your judges cost or what they decide.