Rules live under Manage → Rules.
Rules route, scorers score
This is the distinction worth getting right before you build one. A scorer scores traffic, and owns its own sampling, because what a judge costs is a scorer question. A rule routes traffic, and never scores anything. The consequence people care about: enabling a rule cannot change what your judges cost or what verdicts they produce. Routing is cheap - no LLM call - so the only reason to sample below 100% of whatever matches the filter is queue volume, not the bill. (The dashboard starts a new rule at 10% for exactly that reason; a rule created via the API with nosampleRate runs at 100%.) Dial
the rule down when the queue fills up.
Building one
A rule is a filter, a sample rate, and an action. The filter has typed fields rather than an expression language:- Any trace or errored only
- Contains - text that must appear in the input or output
- Model - exact match
scopeMode/agentIds
and is enforced at run time, but the rule editor doesn’t expose a picker for it yet.)
This is a deliberate limit. A rule whose filter cannot match anything is visibly wrong in the
editor, instead of parsing cleanly into “match nothing” and looking healthy while doing nothing.
The sample rate then applies to whatever survived the filter: 10% of matching traces, not 10%
of all traffic.
Honest activity
Every rule shows eitherfired 24x · last 3 minutes ago or, plainly, never fired. A rule that
has never matched anything looks different from one working hard, because the most common failure
here is not a broken rule - it is a rule that quietly matches nothing while everyone assumes
coverage exists.
Two rules worth having
Sample ordinary traffic
10% of everything → Send to review. This is what keeps judge calibration measured on
normal traffic, not only on what got flagged.
Harvest failures
Errored only → Add to dataset. Real production failures become regression cases. The
expected answer stays blank on purpose: what the agent said is what happened, not what
should have happened.
API
dataset without a datasetId,
webhook without a url - rather than stored as a rule that can never do anything. A rule action
that fails at run time is logged and isolated, so one broken rule never stops the others and never
fails an ingest.
Rules are included in backups.
