Automation Rules

A rule watches incoming traces and routes the ones that match somewhere useful. Three things it can do, one per rule:

Action	What lands where
Send to review	The trace joins the human review queue for a good/bad label.
Add to dataset	The trace becomes a dataset case, with the expected result left blank for a human to write.
Post to webhook	A small JSON payload (Slack-compatible) with the rule name and trace id.

Rules live under Manage → Rules.

Rules route, scorers score

This is the distinction worth getting right before you build one. A scorer scores traffic, and owns its own sampling, because what a judge costs is a scorer question. A rule routes traffic, and never scores anything. The consequence people care about: enabling a rule cannot change what your judges cost or what verdicts they produce. Routing is cheap - no LLM call - so the only reason to sample below 100% of whatever matches the filter is queue volume, not the bill. (The dashboard starts a new rule at 10% for exactly that reason; a rule created via the API with no sampleRate runs at 100%.) Dial the rule down when the queue fills up.

Building one

A rule is a filter, a sample rate, and an action. The filter has typed fields rather than an expression language:

Any trace or errored only
Contains - text that must appear in the input or output
Model - exact match

(An agent scope - all agents, or a chosen few - also exists on the wire as scopeMode/agentIds and is enforced at run time, but the rule editor doesn’t expose a picker for it yet.) This is a deliberate limit. A rule whose filter cannot match anything is visibly wrong in the editor, instead of parsing cleanly into “match nothing” and looking healthy while doing nothing. The sample rate then applies to whatever survived the filter: 10% of matching traces, not 10% of all traffic.

Honest activity

Every rule shows either fired 24x · last 3 minutes ago or, plainly, never fired. A rule that has never matched anything looks different from one working hard, because the most common failure here is not a broken rule - it is a rule that quietly matches nothing while everyone assumes coverage exists.

Two rules worth having

Sample ordinary traffic

10% of everything → Send to review. This is what keeps judge calibration measured on normal traffic, not only on what got flagged.

Harvest failures

Errored only → Add to dataset. Real production failures become regression cases. The expected answer stays blank on purpose: what the agent said is what happened, not what should have happened.

API

curl -X POST "$AGENTX_URL/api/v1/agent-monitoring/rules" \
  -H "x-api-key: $AGENTX_API_KEY" -H "content-type: application/json" \
  -d '{
        "name": "Sample support traffic",
        "action": "review",
        "sampleRate": 0.1,
        "filter": {"status": "any", "contains": "order"}
      }'

An action missing its configuration is refused at creation - dataset without a datasetId, webhook without a url - rather than stored as a rule that can never do anything. A rule action that fails at run time is logged and isolated, so one broken rule never stops the others and never fails an ingest. Rules are included in backups.

​Rules route, scorers score

​Building one

​Honest activity

​Two rules worth having