> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Scorers

> One catalog for everything that can score your traffic - nothing runs until you enable it

A **scorer** is anything that can pass judgment on your agent's traffic. The **Scorers** page is
the single catalog for all of them - one list, filtered by intent rather than split by
implementation, with an inline enable toggle per row. Three kinds live side by side:

| Kind                   | What it is                                                                                                                                                                                                                  | Cost                                        |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| **Template / Pattern** | Deterministic rules - the six [shipped templates](/sdk/monitor) (secrets, PII, injection echo, profanity, refusal, malformed JSON) plus [patterns you author](/monitor/patterns) (phrase, regex, or LLM-judged rubric rows) | Zero LLM cost (except semantic rubric rows) |
| **LLM judge**          | [Continuous judge scoring](/monitor/online-evaluators) against an evaluator config's criteria - per trace, or per session once a conversation goes idle                                                                     | One judge call per sampled trace/session    |
| **Custom**             | [Your own logic](/monitor/custom-evaluators): an external HTTP endpoint returning a verdict, or a code scorer - a Python/JavaScript `handler()` you write, run in-engine                                                    | Whatever your code does                     |

Everything is **opt-in**: a fresh project applies no scorer at all, and the page's footer says so
plainly - nothing is judged until a scorer is enabled. Operational outcomes (trace errors, tool
failures, empty responses) are *not* scorers and need no enabling: the trace itself recorded
them, and they feed the KPI failure metrics directly.

## The list

* **Stat strip**: how many scorers are enabled, how many are firing this week, total signal
  occurrences in the window, and how many are available but off.
* **Filters**: scope pills (All / Enabled / Firing / Off), a kind filter, and search. Rows sort
  enabled-first, then by signal volume.
* **Signals · 7d**: the payoff column - each enabled scorer's signal count for the week with a
  daily sparkline, "None yet" for enabled-but-quiet, "Not running" for disabled.
* **Enabled**: a toggle, right on the row. Built-in templates store this project-wide
  (`enabledBuiltinPatterns` in monitoring defaults); the other kinds carry their own flag.
* **Row actions**: click a row (or its menu) to open the scorer's editor - built-in templates
  open read-only with an "Exactly what it checks" list of their literal rules; LLM judges also
  get **Ratings** (score-over-time) and **Tune judge** (rewrite criteria from calibration
  evidence).

**New scorer** offers the four authorable kinds: a pattern (deterministic rules), an LLM judge
(criteria in plain language + a judge model), a code scorer (your own Python/JavaScript
`handler()`, run in-engine), or an external endpoint.

## First run

Until the first scorer is enabled, the page opens as a starting point instead of a wall of
paused rows: recommended templates to switch on with one click (all zero-LLM-cost, plus the
built-in Session Baseline Judge), or jump straight into authoring your own.

## Where scorer verdicts land

An enabled scorer's hits raise **signals** - deduped triage rows in [Monitor](/sdk/monitor).
Judge scorers additionally record a rating per sampled trace whether or not it crossed the
alert threshold, feeding [Judge Calibration](/monitor/outcomes) against outcomes and
[user feedback](/monitor/user-feedback).

### What moves which metric

Not everything that raises a signal reclassifies the run - build alerting on the right surface:

| Event                                                           | Raises a triage signal | Counts into failureRate / run outcomes                        |
| --------------------------------------------------------------- | ---------------------- | ------------------------------------------------------------- |
| Operational outcome (trace error, tool failure, empty response) | No                     | **Yes** (operational failures)                                |
| Template/pattern scorer hit                                     | Yes                    | **Yes** (scorer detections)                                   |
| LLM judge score below threshold                                 | Yes                    | No - judge scores are evaluator events, tracked per evaluator |
| Code / external scorer hit                                      | Yes                    | No - same evaluator-event semantics as judge scores           |
| User downvote                                                   | Yes                    | No - moves the Downvote rate KPI instead                      |

The practical consequence: an alert built on `failureRate` will see operational failures and
pattern hits, but NOT code-scorer, external-scorer, or judge verdicts - watch those scorers'
own signals and per-scorer event history (or the Scorers page's Signals column) instead.