> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.
# Online Evaluation (Monitor)
> Continuous evaluation of live production traffic - deterministic patterns and LLM-as-judge scoring at ingest, surfaced as triage-ready signals.
**Online evaluation** scores traffic as it happens: every checked trace is evaluated the
moment it arrives, with no dataset and no separate run. Unlike
[offline evaluation](/sdk/evaluations/overview), there is **no ground truth** here - nobody
wrote an expected answer for a live user's question - so verdicts come from LLM-as-judge
scoring and deterministic detectors running against real production behavior, exactly where
your users experience it.
In the dashboard this is the **Monitor** tab. When it finds something, the finding loops back:
a bad trace becomes an offline dataset case in one click, so today's production failure is
tomorrow's regression test.
**Doing it well:**
* **Score the behaviors that hurt in production.** [Online evaluators](/monitor/online-evaluators)
run judge criteria (faithfulness, tool accuracy, goal completion) against sampled live
traces and whole sessions; [patterns](/monitor/patterns) catch the deterministic failures
(errors, empty responses, PII, latency) at zero LLM cost.
* **Tune the sample rate to your volume.** Every judged trace is a real LLM call - start at
`sample_rate=1.0` while traffic is small and you're validating criteria, then dial down as
volume grows. Deterministic patterns stay at 100%: they're free.
* **Let users vote.** [End-user feedback](/monitor/outcomes) (`client.feedback.report`) is the
cheapest evaluation signal you'll ever get - a downvote raises a signal directly, and
[real-world outcomes](/monitor/outcomes) measure how often your judges agreed with reality.
* **Feed findings back into offline tests.** Low-scoring traces (and suspiciously perfect
ones) become [dataset cases](/evaluation/datasets-from-production) - the online/offline loop
is what makes both sides better.
Monitor is AgentX's automatic quality monitoring for production traffic. Every checked trace is evaluated against a set of detectors, and a match becomes a **signal**: a deduped, triage-ready record readable from the dashboard (Governance > Monitor) or straight from the SDK via `client.monitor.signals`.
A **pattern** is a detection rule with a real id, the same way a dataset or an evaluation config has one. Build a pattern once with `client.monitor.patterns.builder(...).publish()`, then reference it by id, or rely on the built-in checks and any patterns your workspace has defined in the dashboard.
This works the same way for an agent built natively in AgentX and for an external agent traced entirely through the Python SDK.
## Two ways to trigger it
They can be combined; neither requires changes to how you already call `tracer.trace(...)` beyond the flags described below.
### Trace-time: `monitor` and `pattern_ids`
Pass `monitor=True` on `tracer.trace(...)` to check that specific trace immediately, with no dashboard setup at all. `pattern_ids` restricts detection to exactly those patterns; omit it to run the full default sweep (built-in checks plus every active pattern) instead.
`monitor=False` is the opposite: it opts that trace out of **every** ingest-time check - patterns, online evaluators, topic classification. Use it for traces produced inside evaluation runs, where the dataset's own judge already scores each case and a second judging pass would only double the bill. Leaving `monitor` unset keeps the engine's normal behavior.
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
pattern = client.monitor.patterns.builder(
name="Promises a refund",
detector_kind="semantic",
semantic_prompt="The response promises a refund.",
severity="high",
).publish()
with client.tracer.trace(
"support-agent", monitor=True, pattern_ids=[pattern.id]
) as span:
span.output = call_llm(query)
```
`pattern_ids` fully defines what's checked when provided: only those named
patterns run, the built-in checks are skipped. This mirrors how
`evaluation_settings_id` fully defines an evaluation's grading config rather
than layering on top of a default. Omit `pattern_ids` (keep just
`monitor=True`) to run the full default sweep instead.
### Dashboard toggle: automatic, every trace from an agent
Enable monitoring once per agent in the dashboard, and every subsequent trace from that agent is checked automatically, with no `monitor=True` needed on any individual call.
1. **Send at least one trace.** The first call to `tracer.trace(...)` for a given agent name auto-creates a reference agent in your workspace. See [Tracing](/sdk/tracing) if you haven't wired this up yet.
2. **Open Governance > Observe > Agents.** Your SDK-traced agent appears in the agent list with an **External** badge, alongside any native agents.
3. **Turn on monitoring.** Enable the agent's monitoring profile and choose a coverage mode: sample a percentage of traffic, or check every trace.
## What gets checked
Three separate streams classify a monitored trace:
**Operational outcomes** - facts the trace itself recorded, always on, never configurable, and
never raising triage signals (they feed the KPI failure metrics and Top failing instead):
| Operational outcome | Recorded when |
| ------------------- | ---------------------------------------------------------------------------------------------------------- |
| Tool/action failure | A recorded tool call reports `success: false` |
| Trace error | `error` was set on the trace (e.g. via `span.set_error(...)`, or an exception inside a decorated function) |
| Empty response | The traced output is empty |
Latency is a distribution metric (Overview's p95 card, straight from the traces), not a failure
classification.
**Scorers** - judgments you opt in to from the [Scorers page](/monitor/scorers), all off by default. Six built-in
template scorers ship with the engine, all zero-LLM-cost; your own template patterns, LLM judge
scorers, and custom endpoint scorers layer on top:
| Built-in template scorer | Flags when the response contains | Severity |
| ------------------------ | ------------------------------------------------------------------------------------------------------------ | -------- |
| Secrets in response | A leaked credential: API keys (OpenAI/AWS/GitHub/Slack), bearer tokens, JWTs, private-key blocks | critical |
| PII in response | Personal data: emails, phone numbers, SSNs, payment cards | high |
| Prompt injection echo | A known jailbreak phrasing repeated in the output ("ignore previous instructions", system-prompt disclosure) | high |
| Profanity in response | Profanity from a small unambiguous wordlist | medium |
| Refusal / non-answer | A deflection ("I can't help with that") - sometimes correct behavior, surfaced for triage | low |
| Malformed JSON response | Output that isn't parseable JSON - enable only for agents whose contract is JSON | medium |
**Human feedback** - your users' votes via `client.feedback.report(trace_id, "down")`, raising a
*Negative user feedback* signal directly (see [User Feedback](/monitor/user-feedback)).
Metric semantics: operational outcomes and template/pattern hits classify the RUN (they move
`failureRate` and the run-outcome breakdown); judge, code, and external scorer verdicts are
evaluator events - they raise signals and keep per-scorer histories without reclassifying the
run. See [what moves which metric](/monitor/scorers#what-moves-which-metric).
## Scorer administration as code
Everything the Scorers page does is scriptable via `client.monitor.scorers` - enable the
shipped templates, deploy code/external scorers, and read their check history:
```python theme={null}
# Templates are opt-in; enable() preserves what's already on and returns the resulting list.
client.monitor.scorers.enable(["pii-in-response", "secrets-in-response"])
client.monitor.scorers.templates() # keys, rules, enabled state
scorer = client.monitor.scorers.create_code(
"Apology overload",
"async def handler(input, output, expected, metadata, trace):\n"
" return 0.0 if str(output).lower().count('sorry') >= 2 else 1.0\n",
language="python", sample_rate=0.5, alert_below=0.5,
)
client.monitor.scorers.dry_run(kind="code", language="python", script="...") # nothing persisted
client.monitor.scorers.events(scorer["_id"], window="24h") # per-check history
client.monitor.scorers.create_external("Policy endpoint", "https://scorer.internal/score")
```
Projects and trace read-back are first-class too: `client.projects.create(name)` returns an
isolated project with its own `apiKey` (the per-run isolation pattern integration suites use),
and `client.traces.get(trace_id)` / `client.traces.list(cursor=...)` read what the tracer wrote.
Pattern scorers you author are rules matched against the response text or the trace: keyword/`contains`, `regex`, or an LLM-judged semantic rubric. Build them via the dashboard (Scorers → **New scorer** → **Pattern scorer** - see [Patterns](/monitor/patterns) for the condition builder, AND/OR/NOR combination, and match targets) or with `client.monitor.patterns.builder(...)`, shown below.
The "Tool/action failure" check reads the `success: false` a recorded tool
call carries. The easiest way to produce that is
`tracer.trace_tool_call(...)`, which captures an escaping exception as a
failed call automatically - see [Tracing](/sdk/tracing#context-manager).
A trace that matches nothing becomes a healthy "info" tally instead, which powers the agent's health-rate percentage.
## `client.monitor.patterns`
```python theme={null}
pattern = client.monitor.patterns.builder(
name="Promises a refund",
detector_kind="semantic",
semantic_prompt="The response promises a refund.",
).publish()
print(pattern.id)
client.monitor.patterns.get(pattern.id) # fetch one
client.monitor.patterns.list() # list every pattern in the workspace
```
### `builder()` parameters
| Parameter | Type | Default | Description |
| --------------------------------- | ------------------- | -------------- | -------------------------------------------------------------------------------- |
| `name` | `str` | required | Pattern display name |
| `description` | `str` | none | Human-readable description |
| `detector_kind` | `str` | `"contains"` | `"contains"`, `"regex"`, or `"semantic"`, selects which field below is used |
| `match_target` | `list[str]` | `["response"]` | Where to look: `"response"`, `"userMessage"`, `"trace"` |
| `match_mode` | `str` | `"any"` | For `detector_kind="contains"`: `"any"` or `"all"` of `include_terms` must match |
| `include_terms` / `exclude_terms` | `list[str]` | `[]` | Phrases to require / exclude, for `detector_kind="contains"` |
| `regex` | `str` | none | Regular expression body, for `detector_kind="regex"` |
| `semantic_prompt` | `str` | none | Rubric an LLM judges the response against, for `detector_kind="semantic"` |
| `severity` | `str` | `"medium"` | `"low"`, `"medium"`, `"high"`, or `"critical"` |
| `polarity` | `str` | `"failure"` | `"failure"` raises a signal to triage; `"proper"` logs a healthy tally instead |
| `enabled` | `bool` | `True` | Whether the pattern is checked at all |
| `sample_rate` | `float` | `1.0` | Fraction of matching traces to actually check, `0.0` to `1.0` |
| `scope_mode` / `agent_ids` | `str` / `list[str]` | `"all"` / `[]` | Restrict this pattern to specific agents instead of the whole workspace |
`publish()` returns a pattern with `.id`, which you pass in `pattern_ids` at trace time.
On the hosted platform, creating a pattern requires a Business or Enterprise
plan (the same entitlement gate as the dashboard's Patterns UI). Self-host has
no plan gates.
## Where signals show up
Matches are deduped by pattern and (usually) by agent, so a recurring issue accumulates occurrences on one signal instead of creating a new row every time. Each signal links back to the trace and tool calls that produced it, so a reviewer can open the exact exchange from the dashboard (Governance > Observe), or read it straight from the SDK.
## `client.monitor.signals`
Read-only: a signal is the system's output from checking traces against patterns, not something you create directly.
```python theme={null}
signals = client.monitor.signals.list(severity="high", limit=20)
for s in signals:
print(s.id, s.severity, s.summary, s.occurrence_count)
signal = client.monitor.signals.get(signals[0].id)
print(signal.recommended_actions)
```
### `list()` parameters
| Parameter | Type | Default | Description |
| ---------- | ----- | ------------------------------ | ------------------------------------------------------------------------------------------------ |
| `polarity` | `str` | server default (failures only) | `"failure"`, `"proper"` (healthy tally), or `"all"` for both |
| `status` | `str` | none | e.g. `"open"`, filter to one status |
| `severity` | `str` | none | `"low"`, `"medium"`, `"high"`, or `"critical"` |
| `agent_id` | `str` | none | Restrict to one agent (matches either the signal's representative agent or its occurrence trail) |
| `limit` | `int` | `50` | Capped at 100 server-side |
`list()`/`get()` both return a signal with `.id`, `.type`, `.severity`, `.polarity`, `.status`, `.summary`, `.pattern_key`, `.occurrence_count`, `.occurrences`, `.recommended_actions`, `.root_cause`, and more, matching the fields shown in the dashboard's triage queue.
## `client.monitor.profile`
Get/update one agent's Monitor coverage settings: coverage mode, sample rate, retention, redaction, and approval policy. Legacy `threshold_overrides` fields still round-trip for wire compatibility, but latency is a KPI metric now (Overview's p95 card, straight from the traces) rather than a detection threshold.
```python theme={null}
profile = client.monitor.profile.get("agent_123")
print(profile.coverage_mode if profile else "never configured, on defaults")
client.monitor.profile.update("agent_123", coverage_mode="all", sample_rate=0.5)
```
`get()` returns `None` when the agent has never been configured (still on platform defaults). `update()` upserts and only changes the fields you pass:
| Parameter | Type | Description |
| ------------------------------------------------------ | ---------------- | ------------------------------------------------------------ |
| `enabled` | `bool` | Turn Monitor on/off for this agent |
| `failure_detection_enabled` / `info_detection_enabled` | `bool` | Opt a whole detection category out |
| `coverage_mode` | `str` | `"all"` (every trace) or `"sampled"` |
| `sample_rate` | `float` | Fraction of traffic monitored when `coverage_mode="sampled"` |
| `channels` | `list[str]` | Notification channels |
| `dataset_id` | `str` | Evaluation dataset this agent's signals feed into |
| `threshold_overrides` | `dict` | Per-check threshold overrides, e.g. `{"latencyMs": 15000}` |
| `retention_days` | `int` | How long monitored traces are kept |
| `redaction_mode` | `str` | `"none"`, `"standard"`, or `"strict"` |
| `approval_policy` | `dict[str, str]` | Per-action approval mode for autotune actions |
## Self-host extras
A [self-hosted](/self-host/overview) instance adds two Monitor surfaces the hosted platform doesn't have yet, both reachable from this same SDK:
* **Online evaluators** (`client.monitor.online_evaluators`) - a real LLM judge scoring a sample of live traffic continuously, per trace or per whole session (`scope="session"`, judged automatically once the conversation goes idle). See [Online evaluators](/monitor/online-evaluators).
* **Outcome reports and user feedback** (`client.outcomes.report(...)` / `client.feedback.report(...)`) - record what actually happened after the fact (a reopened ticket, an end user's thumbs-down) against a trace; negative feedback raises a signal directly, and both streams feed the dashboard's Judge Calibration card. See [Outcomes & Judge Calibration](/monitor/outcomes).
Send the traces this feature monitors
The condition builder, match targets, and where signals go