> ## Documentation Index > Fetch the complete documentation index at: https://developers.agentx.so/llms.txt > Use this file to discover all available pages before exploring further. # Alert Rules > Page Slack, Teams, PagerDuty, email, or a webhook when a KPI crosses a threshold Three things in AgentX notify, and it helps to know which one you are reaching for: * A **scorer's alert threshold** fires per verdict - one trace the judge rated badly. * An **automation rule** routes per trace - one matching trace into review, a dataset, or a webhook. * An **alert rule** watches an **aggregate** over a sliding window - failure rate, p95 latency, spend, judge failures, traffic volume - and pages an on-call channel when it crosses a threshold. This is the page an ops team used to build by hand from raw webhook payloads. Alert rules live under **Manage → Rules**, below the automation rules. ## What you can alert on | Metric | Wire name | Unit | Computed from | | ------------------ | ------------------ | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Failure rate | `failureRate` | fraction (0.10 = 10%) | Monitored runs with a failure-polarity detection (an operational error or a failing pattern) over all monitored runs in the window - the same classification as the Overview's KPI cards. Judge, scorer-group, and custom-scorer verdicts are ratings, not run outcomes, and do not count | | Tool failure rate | `toolFailureRate` | fraction | Monitored runs with at least one failed tool call | | p95 latency | `p95LatencyMs` | milliseconds | 95th percentile of root-trace latency across production traffic | | Estimated LLM cost | `estimatedCostUsd` | USD | Token usage on root production traces priced against the [model catalog](/evaluation/model-portability#the-pricing-catalog); unpriced models contribute \$0 | | Judge failures | `judgeFailures` | count | Live judge calls that returned no usable verdict (provider errors, missing keys, parse failures) | | Trace count | `traceCount` | count | Root production traces in the window - alert **below** a floor to catch traffic silently stopping | A rate with nothing in its window is "no data", never a breach: an empty deployment does not page you for a 0% failure rate, and "no traffic at all" is what a `traceCount` floor is for. Windows run from 1 minute to 7 days (the editor offers 5m, 15m, 1h, 6h, 24h); a rule can watch every agent in the project or one agent. Evaluation-run traffic is excluded from every metric, and on the ClickHouse tier the trace-derived metrics read spans from ClickHouse directly. ## Lifecycle Rules are evaluated once a minute by a background sweep (one evaluator per tick across replicas that share a database). The lifecycle is the one pagers expect: 1. **Triggered** - the first evaluation that breaches sends one notification and sets the rule to `firing`. 2. **Repeat** - while it keeps breaching, a reminder goes out every `cooldownMinutes` (default 60), never more often. 3. **Resolved** - the first evaluation back inside the threshold sends a resolve notification (PagerDuty closes the incident it opened, with the same dedup key) and the rule returns to `ok`. Every notification carries an `incidentKey` that is stable for one firing episode (the trigger, its repeats, and its resolve) and fresh for the next, so a new incident is never deduplicated into an old one. Editing a rule's metric, operator, threshold, window, or agent resets it to `ok`, and so does pausing it (`enabled: false`) - a re-thresholded rule never sits "firing" on a value it would no longer breach, and a rule re-enabled mid-incident pages a fresh "triggered". If the rule was firing at that moment, its channels get a final resolved notification first (PagerDuty closes the incident), and deleting a firing rule does the same before it goes. Each rule keeps its last 200 notifications as history, with every channel's delivery result, so "did the page actually go out?" has an answer. ## Channels A rule pages up to 5 channels. Each channel is a `{ kind, target }` pair: | Kind | Target | What is sent | | ----------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `slack` | Incoming-webhook URL | A Block Kit message: status header, summary, and a fact grid (metric, current value, threshold, window, severity, agent) | | `teams` | Incoming-webhook or Workflows URL | An Adaptive Card in the `attachments` envelope, accepted by both the classic connector and Workflows | | `pagerduty` | Events API v2 integration (routing) key | A `trigger` event on firing and repeat, a `resolve` event on recovery, deduplicated per firing episode (the `incidentKey`); severity maps low/medium/high/critical to info/warning/error/critical | | `email` | Address | A plain-text email through the engine's mailer (`AGENTX_SMTP_URL`, `AGENTX_RESEND_API_KEY`, or `AGENTX_EMAIL_DEBUG_DIR` - see [Configuration](/self-host/configuration)); without one the channel reports a delivery failure | | `webhook` | Any URL | The full structured notification as JSON, plus a top-level `text` so a Slack-style receiver still shows something readable | Channel targets are validated when the rule is saved: a URL must pass the engine's egress guard (http(s) only, cloud metadata endpoints refused, private targets refused on multi-tenant deployments), an email must be an address, a PagerDuty key must look like one. A channel that could never deliver is refused with a 400 instead of failing silently on the first real incident. The threshold is checked against the metric the same way: a count (`judgeFailures`, `traceCount`) must be a whole number and a rate must be between 0 and 1. PagerDuty keys are stored once and shown masked on every read; saving the rule with the mask in place keeps the stored key. The generic webhook body: ```json theme={null} { "text": "[AgentX Alert] FIRING: Support failures - Failure rate above 10.0% over the last 15m - currently 23.4%.", "event": "alert_rule", "kind": "triggered", "status": "FIRING", "ruleId": "oSzjCrCgfc8ZPOb0kOaM7", "ruleName": "Support failures", "severity": "high", "metric": "failureRate", "metricLabel": "Failure rate", "operator": "gt", "threshold": 0.1, "thresholdLabel": "10.0%", "value": 0.234, "valueLabel": "23.4%", "windowMinutes": 15, "windowLabel": "15m", "agentId": null, "agentName": null, "incidentKey": "agentx-alert-oSzjCrCgfc8ZPOb0kOaM7-1757962800000", "condition": "Failure rate above 10.0% over the last 15m", "title": "[AgentX Alert] FIRING: Support failures", "summary": "Failure rate above 10.0% over the last 15m - currently 23.4%.", "at": "2026-09-15T19:05:12.140Z" } ``` `kind` is `triggered`, `repeat`, `resolved`, or `test`; `status` is `FIRING`, `RESOLVED`, or `TEST`. `agentName` is the scoped agent's display name (`null` for an all-agents rule), and `incidentKey` is the firing episode's identity described under Lifecycle. A delivery that a channel refuses is recorded as `HTTP ` in the rule's history - never the receiver's response body. ## In the dashboard The editor shows the metric's **current value** for the draft's window and agent as you build the rule - the same computation the sweep will run - so a threshold is set against real numbers. Each rule row shows its state (`ok`, `firing`, or `paused`), the live value, and honest activity ("never fired", or the incident count and when it last fired). **Send test** delivers a test notification to every channel with the live value and reports each channel's result, so a mistyped hook shows up now rather than during an incident; **History** opens the per-channel delivery log. ## From the SDK ```python theme={null} from agentx.monitor.alert_rules import slack, pagerduty, email alerts = client.monitor.alert_rules rule = alerts.create( "Support failure rate above 10%", metric="failureRate", operator="gt", # "gt" = above, "lt" = below threshold=0.10, # rates are fractions; latency is ms; cost is USD window_minutes=15, severity="high", cooldown_minutes=60, agent_id=None, # or one agent's id channels=[ slack("https://hooks.slack.com/services/T000/B000/XXXX"), pagerduty("R0123456789abcdef0123456789abcdef"), email("oncall@example.com"), ], ) alerts.preview("p95LatencyMs", window_minutes=15) # {"value": 812, "valueLabel": "812 ms", ...} event = alerts.test(rule.id) # a TEST page to every channel print(event.delivered, event["deliveries"]) # per-channel ok / status / error for e in alerts.events(rule.id): # history, newest first print(e.kind, e.delivered) alerts.update(rule.id, threshold=0.15, cooldown_minutes=30) alerts.run_sweep() # evaluate this project's rules now alerts.delete(rule.id) ``` `create` and `update` refuse an unknown metric, operator, or channel kind locally, and a channel the engine cannot deliver to comes back as a 400 with the reason. Deleting a rule drops its history; if it was firing, its channels get a final resolved notification first, so the PagerDuty incident it opened is closed with it. ## REST All routes live under `/api/v1/agent-monitoring/alert-rules` and take the project API key: | Method | Path | Purpose | | ------------------------ | -------------------------------- | ------------------------------------------------------------------------------------- | | `GET` | `/alert-rules` | List rules with a `{ total, enabled, firing }` summary | | `POST` | `/alert-rules` | Create (`409` past 50 rules per project, `400` for an undeliverable channel) | | `GET` / `PUT` / `DELETE` | `/alert-rules/:id` | Read, sparse update, delete | | `GET` | `/alert-rules/:id/events?limit=` | Notification history with per-channel deliveries | | `POST` | `/alert-rules/:id/test` | Send a test notification; returns the recorded event and `delivered` | | `POST` | `/alert-rules/preview` | `{ metric, windowMinutes, agentId? }` → the metric's live value | | `POST` | `/alert-rules/sweep/run` | Evaluate this project's rules now (the background sweep runs every minute regardless) | `AGENTX_ALERT_SWEEP=false` disables the background evaluator entirely; the manual route still works. Alert rules and their history ride along in [backups](/self-host/backup) as the `alert-rules` and `alert-events` entities, with hook URLs and PagerDuty keys redacted. The runnable walkthrough is `sample-scripts/monitor_ops/08_alert_rules_and_streaming.py`: it stands up a local Slack stand-in, drives a failure-rate rule through triggered → held → resolved with real traffic, and sends a test page.