Alert Rules

Three things in AgentX notify, and it helps to know which one you are reaching for:

A scorer’s alert threshold fires per verdict - one trace the judge rated badly.
An automation rule routes per trace - one matching trace into review, a dataset, or a webhook.
An alert rule watches an aggregate over a sliding window - failure rate, p95 latency, spend, judge failures, traffic volume - and pages an on-call channel when it crosses a threshold. This is the page an ops team used to build by hand from raw webhook payloads.

Alert rules live under Manage → Rules, below the automation rules.

What you can alert on

Metric	Wire name	Unit	Computed from
Failure rate	`failureRate`	fraction (0.10 = 10%)	Monitored runs with a failure-polarity detection (an operational error or a failing pattern) over all monitored runs in the window - the same classification as the Overview’s KPI cards. Judge, scorer-group, and custom-scorer verdicts are ratings, not run outcomes, and do not count
Tool failure rate	`toolFailureRate`	fraction	Monitored runs with at least one failed tool call
p95 latency	`p95LatencyMs`	milliseconds	95th percentile of root-trace latency across production traffic
Estimated LLM cost	`estimatedCostUsd`	USD	Token usage on root production traces priced against the model catalog; unpriced models contribute $0
Judge failures	`judgeFailures`	count	Live judge calls that returned no usable verdict (provider errors, missing keys, parse failures)
Trace count	`traceCount`	count	Root production traces in the window - alert below a floor to catch traffic silently stopping

A rate with nothing in its window is “no data”, never a breach: an empty deployment does not page you for a 0% failure rate, and “no traffic at all” is what a traceCount floor is for. Windows run from 1 minute to 7 days (the editor offers 5m, 15m, 1h, 6h, 24h); a rule can watch every agent in the project or one agent. Evaluation-run traffic is excluded from every metric, and on the ClickHouse tier the trace-derived metrics read spans from ClickHouse directly.

Lifecycle

Rules are evaluated once a minute by a background sweep (one evaluator per tick across replicas that share a database). The lifecycle is the one pagers expect:

Triggered - the first evaluation that breaches sends one notification and sets the rule to firing.
Repeat - while it keeps breaching, a reminder goes out every cooldownMinutes (default 60), never more often.
Resolved - the first evaluation back inside the threshold sends a resolve notification (PagerDuty closes the incident it opened, with the same dedup key) and the rule returns to ok.

Every notification carries an incidentKey that is stable for one firing episode (the trigger, its repeats, and its resolve) and fresh for the next, so a new incident is never deduplicated into an old one. Editing a rule’s metric, operator, threshold, window, or agent resets it to ok, and so does pausing it (enabled: false) - a re-thresholded rule never sits “firing” on a value it would no longer breach, and a rule re-enabled mid-incident pages a fresh “triggered”. If the rule was firing at that moment, its channels get a final resolved notification first (PagerDuty closes the incident), and deleting a firing rule does the same before it goes. Each rule keeps its last 200 notifications as history, with every channel’s delivery result, so “did the page actually go out?” has an answer.

Channels

A rule pages up to 5 channels. Each channel is a { kind, target } pair:

Kind	Target	What is sent
`slack`	Incoming-webhook URL	A Block Kit message: status header, summary, and a fact grid (metric, current value, threshold, window, severity, agent)
`teams`	Incoming-webhook or Workflows URL	An Adaptive Card in the `attachments` envelope, accepted by both the classic connector and Workflows
`pagerduty`	Events API v2 integration (routing) key	A `trigger` event on firing and repeat, a `resolve` event on recovery, deduplicated per firing episode (the `incidentKey`); severity maps low/medium/high/critical to info/warning/error/critical
`email`	Address	A plain-text email through the engine’s mailer (`AGENTX_SMTP_URL`, `AGENTX_RESEND_API_KEY`, or `AGENTX_EMAIL_DEBUG_DIR` - see Configuration); without one the channel reports a delivery failure
`webhook`	Any URL	The full structured notification as JSON, plus a top-level `text` so a Slack-style receiver still shows something readable

Channel targets are validated when the rule is saved: a URL must pass the engine’s egress guard (http(s) only, cloud metadata endpoints refused, private targets refused on multi-tenant deployments), an email must be an address, a PagerDuty key must look like one. A channel that could never deliver is refused with a 400 instead of failing silently on the first real incident. The threshold is checked against the metric the same way: a count (judgeFailures, traceCount) must be a whole number and a rate must be between 0 and 1. PagerDuty keys are stored once and shown masked on every read; saving the rule with the mask in place keeps the stored key. The generic webhook body:

{
  "text": "[AgentX Alert] FIRING: Support failures - Failure rate above 10.0% over the last 15m - currently 23.4%.",
  "event": "alert_rule",
  "kind": "triggered",
  "status": "FIRING",
  "ruleId": "oSzjCrCgfc8ZPOb0kOaM7",
  "ruleName": "Support failures",
  "severity": "high",
  "metric": "failureRate",
  "metricLabel": "Failure rate",
  "operator": "gt",
  "threshold": 0.1,
  "thresholdLabel": "10.0%",
  "value": 0.234,
  "valueLabel": "23.4%",
  "windowMinutes": 15,
  "windowLabel": "15m",
  "agentId": null,
  "agentName": null,
  "incidentKey": "agentx-alert-oSzjCrCgfc8ZPOb0kOaM7-1757962800000",
  "condition": "Failure rate above 10.0% over the last 15m",
  "title": "[AgentX Alert] FIRING: Support failures",
  "summary": "Failure rate above 10.0% over the last 15m - currently 23.4%.",
  "at": "2026-09-15T19:05:12.140Z"
}

kind is triggered, repeat, resolved, or test; status is FIRING, RESOLVED, or TEST. agentName is the scoped agent’s display name (null for an all-agents rule), and incidentKey is the firing episode’s identity described under Lifecycle. A delivery that a channel refuses is recorded as HTTP <status> in the rule’s history - never the receiver’s response body.

In the dashboard

The editor shows the metric’s current value for the draft’s window and agent as you build the rule - the same computation the sweep will run - so a threshold is set against real numbers. Each rule row shows its state (ok, firing, or paused), the live value, and honest activity (“never fired”, or the incident count and when it last fired). Send test delivers a test notification to every channel with the live value and reports each channel’s result, so a mistyped hook shows up now rather than during an incident; History opens the per-channel delivery log.

From the SDK

from agentx.monitor.alert_rules import slack, pagerduty, email

alerts = client.monitor.alert_rules

rule = alerts.create(
    "Support failure rate above 10%",
    metric="failureRate",
    operator="gt",          # "gt" = above, "lt" = below
    threshold=0.10,         # rates are fractions; latency is ms; cost is USD
    window_minutes=15,
    severity="high",
    cooldown_minutes=60,
    agent_id=None,          # or one agent's id
    channels=[
        slack("https://hooks.slack.com/services/T000/B000/XXXX"),
        pagerduty("R0123456789abcdef0123456789abcdef"),
        email("oncall@example.com"),
    ],
)

alerts.preview("p95LatencyMs", window_minutes=15)   # {"value": 812, "valueLabel": "812 ms", ...}
event = alerts.test(rule.id)                          # a TEST page to every channel
print(event.delivered, event["deliveries"])           # per-channel ok / status / error

for e in alerts.events(rule.id):                      # history, newest first
    print(e.kind, e.delivered)

alerts.update(rule.id, threshold=0.15, cooldown_minutes=30)
alerts.run_sweep()                                    # evaluate this project's rules now
alerts.delete(rule.id)

create and update refuse an unknown metric, operator, or channel kind locally, and a channel the engine cannot deliver to comes back as a 400 with the reason. Deleting a rule drops its history; if it was firing, its channels get a final resolved notification first, so the PagerDuty incident it opened is closed with it.

REST

All routes live under /api/v1/agent-monitoring/alert-rules and take the project API key:

Method	Path	Purpose
`GET`	`/alert-rules`	List rules with a `{ total, enabled, firing }` summary
`POST`	`/alert-rules`	Create (`409` past 50 rules per project, `400` for an undeliverable channel)
`GET` / `PUT` / `DELETE`	`/alert-rules/:id`	Read, sparse update, delete
`GET`	`/alert-rules/:id/events?limit=`	Notification history with per-channel deliveries
`POST`	`/alert-rules/:id/test`	Send a test notification; returns the recorded event and `delivered`
`POST`	`/alert-rules/preview`	`{ metric, windowMinutes, agentId? }` → the metric’s live value
`POST`	`/alert-rules/sweep/run`	Evaluate this project’s rules now (the background sweep runs every minute regardless)

AGENTX_ALERT_SWEEP=false disables the background evaluator entirely; the manual route still works. Alert rules and their history ride along in backups as the alert-rules and alert-events entities, with hook URLs and PagerDuty keys redacted. The runnable walkthrough is sample-scripts/monitor_ops/08_alert_rules_and_streaming.py: it stands up a local Slack stand-in, drives a failure-rate rule through triggered → held → resolved with real traffic, and sends a test page.

The Scorer Catalog

Judge Calibration & Tuning

Live Traffic Scoring

What you can alert on

Lifecycle

Channels

In the dashboard

From the SDK

REST

​What you can alert on

​Lifecycle

​Channels

​In the dashboard

​From the SDK

​REST

What you can alert on

Lifecycle

Channels

In the dashboard

From the SDK

REST