Skip to main content
A rule watches incoming traces and routes the ones that match somewhere useful. Three things it can do, one per rule: Rules live under Manage → Rules.

​
Rules route, scorers score

This is the distinction worth getting right before you build one. A scorer scores traffic, and owns its own sampling, because what a judge costs is a scorer question. A rule routes traffic, and never scores anything. (A third thing notifies on aggregates rather than on single traces - failure rate, p95 latency, spend - and pages an on-call channel: that is an alert rule, which lives on this same page below the routing rules.) The consequence people care about: enabling a rule cannot change what your judges cost or what verdicts they produce. Routing is cheap - no LLM call - so the only reason to sample below 100% of whatever matches the filter is queue volume, not the bill. (The dashboard starts a new rule at 10% for exactly that reason; a rule created via the API with no sampleRate runs at 100%.) Dial the rule down when the queue fills up.

​
Building one

A rule is a filter, a sample rate, and an action. The filter has typed fields rather than an expression language:
  • Any trace or errored only
  • Contains - text that must appear in the input or output
  • Model - exact match
(An agent scope - all agents, or a chosen few - also exists on the wire as scopeMode/agentIds and is enforced at run time, but the rule editor doesn’t expose a picker for it yet.) This is a deliberate limit. A rule whose filter cannot match anything is visibly wrong in the editor, instead of parsing cleanly into “match nothing” and looking healthy while doing nothing. The sample rate then applies to whatever survived the filter: 10% of matching traces, not 10% of all traffic.

​
Honest activity

Every rule shows either fired 24x · last 3 minutes ago or, plainly, never fired. A rule that has never matched anything looks different from one working hard, because the most common failure here is not a broken rule - it is a rule that quietly matches nothing while everyone assumes coverage exists.

​
What a rule never sees

Two kinds of traffic skip rules entirely at ingest: traces sent with monitor=False, and traces with source="eval-run" (unless they explicitly set monitor=True) - the same opt-outs that skip every other ingest-time check. And two matches deflect silently instead of firing. A dataset action does nothing when the case would be a duplicate or the dataset is already at its case cap; a review action does nothing when the trace is already queued or the queue is at its 200-item pending cap. Neither increments the rule’s fired count - so never fired can also mean “always deflected”, not just “never matched”. The deflections are logged engine-side.

​
Two rules worth having

Sample ordinary traffic

10% of everything → Send to review. This is what keeps judge calibration measured on normal traffic, not only on what got flagged.

Harvest failures

Errored only → Add to dataset. Real production failures become regression cases. The expected answer stays blank on purpose: what the agent said is what happened, not what should have happened.

​
API

Update and delete use the same path with the rule’s id: PUT /agent-monitoring/rules/:id (partial - only the fields you send change) and DELETE /agent-monitoring/rules/:id. The Python SDK wraps all of it:
An action missing its configuration is refused at creation - dataset without an action_config datasetId, webhook without a url - rather than stored as a rule that can never do anything. A rule action that fails at run time is logged and isolated, so one broken rule never stops the others and never fails an ingest. Rules are included in backups.