Scorers

A scorer is anything that can pass judgment on your agent’s traffic. The Scorers page is the single catalog for all of them - one list, filtered by intent rather than split by implementation, with an inline enable toggle per row. Three kinds live side by side:

Kind	What it is	Cost
Template / Pattern	Deterministic rules - the six shipped templates (secrets, PII, injection echo, profanity, refusal, malformed JSON) plus patterns you author (phrase, regex, or LLM-judged rubric rows)	Zero LLM cost (except semantic rubric rows)
LLM judge	Continuous judge scoring against an evaluator config’s criteria - per trace, or per session once a conversation goes idle	One judge call per sampled trace/session
Custom	Your own logic: an external HTTP endpoint returning a verdict, or a code scorer - a Python/JavaScript `handler()` you write, run in-engine	Whatever your code does

Everything is opt-in: a fresh project applies no scorer at all, and the page’s footer says so plainly - nothing is judged until a scorer is enabled. Operational outcomes (trace errors, tool failures, empty responses) are not scorers and need no enabling: the trace itself recorded them, and they feed the KPI failure metrics directly.

The list

Stat strip: how many scorers are enabled, how many are firing this week, total signal occurrences in the window, and how many are available but off.
Filters: scope pills (All / Enabled / Firing / Off), a kind filter, and search. Rows sort enabled-first, then by signal volume.
Signals · 7d: the payoff column - each enabled scorer’s signal count for the week with a daily sparkline, “None yet” for enabled-but-quiet, “Not running” for disabled.
Enabled: a toggle, right on the row. Built-in templates store this project-wide (enabledBuiltinPatterns in monitoring defaults); the other kinds carry their own flag.
Row actions: click a row (or its menu) to open the scorer’s editor - built-in templates open read-only with an “Exactly what it checks” list of their literal rules; LLM judges also get Ratings (score-over-time) and Tune judge (rewrite criteria from calibration evidence).

New scorer offers the four authorable kinds: a pattern (deterministic rules), an LLM judge (criteria in plain language + a judge model), a code scorer (your own Python/JavaScript handler(), run in-engine), or an external endpoint.

First run

Until the first scorer is enabled, the page opens as a starting point instead of a wall of paused rows: recommended templates to switch on with one click (all zero-LLM-cost, plus the built-in Session Baseline Judge), or jump straight into authoring your own.

Where scorer verdicts land

An enabled scorer’s hits raise signals - deduped triage rows in Monitor. Judge scorers additionally record a rating per sampled trace whether or not it crossed the alert threshold, feeding Judge Calibration against outcomes and user feedback.

What moves which metric

Not everything that raises a signal reclassifies the run - build alerting on the right surface:

Event	Raises a triage signal	Counts into failureRate / run outcomes
Operational outcome (trace error, tool failure, empty response)	No	Yes (operational failures)
Template/pattern scorer hit	Yes	Yes (scorer detections)
LLM judge score below threshold	Yes	No - judge scores are evaluator events, tracked per evaluator
Code / external scorer hit	Yes	No - same evaluator-event semantics as judge scores
User downvote	Yes	No - moves the Downvote rate KPI instead

The practical consequence: an alert built on failureRate will see operational failures and pattern hits, but NOT code-scorer, external-scorer, or judge verdicts - watch those scorers’ own signals and per-scorer event history (or the Scorers page’s Signals column) instead.

Live Traffic Scoring

Ground Truth & Calibration

The list

First run

Where scorer verdicts land

What moves which metric

​The list

​First run

​Where scorer verdicts land

​What moves which metric

The list

First run

Where scorer verdicts land

What moves which metric