Online Evaluation (Monitor)

Online evaluation scores traffic as it happens: every checked trace is evaluated the moment it arrives, with no dataset and no separate run. Unlike offline evaluation, there is no ground truth here - nobody wrote an expected answer for a live user’s question - so verdicts come from LLM-as-judge scoring and deterministic detectors running against real production behavior, exactly where your users experience it. In the dashboard this is the Monitor tab. When it finds something, the finding loops back: a bad trace becomes an offline dataset case in one click, so today’s production failure is tomorrow’s regression test. Doing it well:

Score the behaviors that hurt in production. Online evaluators run judge criteria (faithfulness, tool accuracy, goal completion) against sampled live traces and whole sessions; patterns catch the deterministic failures (errors, empty responses, PII, latency) at zero LLM cost.
Tune the sample rate to your volume. Every judged trace is a real LLM call - start at sample_rate=1.0 while traffic is small and you’re validating criteria, then dial down as volume grows. Deterministic patterns stay at 100%: they’re free.
Let users vote. End-user feedback (client.feedback.report) is the cheapest evaluation signal you’ll ever get - a downvote raises a signal directly, and real-world outcomes measure how often your judges agreed with reality.
Feed findings back into offline tests. Low-scoring traces (and suspiciously perfect ones) become dataset cases - the online/offline loop is what makes both sides better.

Monitor is AgentX’s automatic quality monitoring for production traffic. Every checked trace is evaluated against a set of detectors, and a match becomes a signal: a deduped, triage-ready record readable from the dashboard (Governance > Monitor) or straight from the SDK via client.monitor.signals. A pattern is a detection rule with a real id, the same way a dataset or an evaluation config has one. Build a pattern once with client.monitor.patterns.builder(...).publish(), then reference it by id, or rely on the built-in checks and any patterns your workspace has defined in the dashboard. This works the same way for an agent built natively in AgentX and for an external agent traced entirely through the Python SDK.

Two ways to trigger it

They can be combined; neither requires changes to how you already call tracer.trace(...) beyond the flags described below.

Trace-time: `monitor` and `pattern_ids`

Pass monitor=True on tracer.trace(...) to check that specific trace immediately, with no dashboard setup at all. pattern_ids restricts detection to exactly those patterns; omit it to run the full default sweep (built-in checks plus every active pattern) instead. monitor=False is the opposite: it opts that trace out of every ingest-time check - patterns, online evaluators, topic classification. Use it for traces produced inside evaluation runs, where the dataset’s own judge already scores each case and a second judging pass would only double the bill. Leaving monitor unset keeps the engine’s normal behavior.

from agentx import AgentX

client = AgentX.from_env()

pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
    severity="high",
).publish()

with client.tracer.trace(
    "support-agent", monitor=True, pattern_ids=[pattern.id]
) as span:
    span.output = call_llm(query)

pattern_ids fully defines what’s checked when provided: only those named patterns run, the built-in checks are skipped. This mirrors how evaluation_settings_id fully defines an evaluation’s grading config rather than layering on top of a default. Omit pattern_ids (keep just monitor=True) to run the full default sweep instead.

Dashboard toggle: automatic, every trace from an agent

Enable monitoring once per agent in the dashboard, and every subsequent trace from that agent is checked automatically, with no monitor=True needed on any individual call.

Send at least one trace. The first call to tracer.trace(...) for a given agent name auto-creates a reference agent in your workspace. See Tracing if you haven’t wired this up yet.
Open Governance > Observe > Agents. Your SDK-traced agent appears in the agent list with an External badge, alongside any native agents.
Turn on monitoring. Enable the agent’s monitoring profile and choose a coverage mode: sample a percentage of traffic, or check every trace.

What gets checked

Three separate streams classify a monitored trace: Operational outcomes - facts the trace itself recorded, always on, never configurable, and never raising triage signals (they feed the KPI failure metrics and Top failing instead):

Operational outcome	Recorded when
Tool/action failure	A recorded tool call reports `success: false`
Trace error	`error` was set on the trace (e.g. via `span.set_error(...)`, or an exception inside a decorated function)
Empty response	The traced output is empty

Latency is a distribution metric (Overview’s p95 card, straight from the traces), not a failure classification. Scorers - judgments you opt in to from the Scorers page, all off by default. Six built-in template scorers ship with the engine, all zero-LLM-cost; your own template patterns, LLM judge scorers, and custom endpoint scorers layer on top:

Built-in template scorer	Flags when the response contains	Severity
Secrets in response	A leaked credential: API keys (OpenAI/AWS/GitHub/Slack), bearer tokens, JWTs, private-key blocks	critical
PII in response	Personal data: emails, phone numbers, SSNs, payment cards	high
Prompt injection echo	A known jailbreak phrasing repeated in the output (“ignore previous instructions”, system-prompt disclosure)	high
Profanity in response	Profanity from a small unambiguous wordlist	medium
Refusal / non-answer	A deflection (“I can’t help with that”) - sometimes correct behavior, surfaced for triage	low
Malformed JSON response	Output that isn’t parseable JSON - enable only for agents whose contract is JSON	medium

Human feedback - your users’ votes via client.feedback.report(trace_id, "down"), raising a Negative user feedback signal directly (see User Feedback).

Metric semantics: operational outcomes and template/pattern hits classify the RUN (they move failureRate and the run-outcome breakdown); judge, code, and external scorer verdicts are evaluator events - they raise signals and keep per-scorer histories without reclassifying the run. See what moves which metric.

Scorer administration as code

Everything the Scorers page does is scriptable via client.monitor.scorers - enable the shipped templates, deploy code/external scorers, and read their check history:

# Templates are opt-in; enable() preserves what's already on and returns the resulting list.
client.monitor.scorers.enable(["pii-in-response", "secrets-in-response"])
client.monitor.scorers.templates()          # keys, rules, enabled state

scorer = client.monitor.scorers.create_code(
    "Apology overload",
    "async def handler(input, output, expected, metadata, trace):\n"
    "    return 0.0 if str(output).lower().count('sorry') >= 2 else 1.0\n",
    language="python", sample_rate=0.5, alert_below=0.5,
)
client.monitor.scorers.dry_run(kind="code", language="python", script="...")  # nothing persisted
client.monitor.scorers.events(scorer["_id"], window="24h")  # per-check history
client.monitor.scorers.create_external("Policy endpoint", "https://scorer.internal/score")

Projects and trace read-back are first-class too: client.projects.create(name) returns an isolated project with its own apiKey (the per-run isolation pattern integration suites use), and client.traces.get(trace_id) / client.traces.list(cursor=...) read what the tracer wrote. Pattern scorers you author are rules matched against the response text or the trace: keyword/contains, regex, or an LLM-judged semantic rubric. Build them via the dashboard (Scorers → New scorer → Pattern scorer - see Patterns for the condition builder, AND/OR/NOR combination, and match targets) or with client.monitor.patterns.builder(...), shown below.

The “Tool/action failure” check reads the success: false a recorded tool call carries. The easiest way to produce that is tracer.trace_tool_call(...), which captures an escaping exception as a failed call automatically - see Tracing.

A trace that matches nothing becomes a healthy “info” tally instead, which powers the agent’s health-rate percentage.

`client.monitor.patterns`

pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
).publish()

print(pattern.id)

client.monitor.patterns.get(pattern.id)   # fetch one
client.monitor.patterns.list()            # list every pattern in the workspace

`builder()` parameters

Parameter	Type	Default	Description
`name`	`str`	required	Pattern display name
`description`	`str`	none	Human-readable description
`detector_kind`	`str`	`"contains"`	`"contains"`, `"regex"`, or `"semantic"`, selects which field below is used
`match_target`	`list[str]`	`["response"]`	Where to look: `"response"`, `"userMessage"`, `"trace"`
`match_mode`	`str`	`"any"`	For `detector_kind="contains"`: `"any"` or `"all"` of `include_terms` must match
`include_terms` / `exclude_terms`	`list[str]`	`[]`	Phrases to require / exclude, for `detector_kind="contains"`
`regex`	`str`	none	Regular expression body, for `detector_kind="regex"`
`semantic_prompt`	`str`	none	Rubric an LLM judges the response against, for `detector_kind="semantic"`
`severity`	`str`	`"medium"`	`"low"`, `"medium"`, `"high"`, or `"critical"`
`polarity`	`str`	`"failure"`	`"failure"` raises a signal to triage; `"proper"` logs a healthy tally instead
`enabled`	`bool`	`True`	Whether the pattern is checked at all
`sample_rate`	`float`	`1.0`	Fraction of matching traces to actually check, `0.0` to `1.0`
`scope_mode` / `agent_ids`	`str` / `list[str]`	`"all"` / `[]`	Restrict this pattern to specific agents instead of the whole workspace

publish() returns a pattern with .id, which you pass in pattern_ids at trace time.

On the hosted platform, creating a pattern requires a Business or Enterprise plan (the same entitlement gate as the dashboard’s Patterns UI). Self-host has no plan gates.

Where signals show up

Matches are deduped by pattern and (usually) by agent, so a recurring issue accumulates occurrences on one signal instead of creating a new row every time. Each signal links back to the trace and tool calls that produced it, so a reviewer can open the exact exchange from the dashboard (Governance > Observe), or read it straight from the SDK.

`client.monitor.signals`

Read-only: a signal is the system’s output from checking traces against patterns, not something you create directly.

signals = client.monitor.signals.list(severity="high", limit=20)
for s in signals:
    print(s.id, s.severity, s.summary, s.occurrence_count)

signal = client.monitor.signals.get(signals[0].id)
print(signal.recommended_actions)

`list()` parameters

Parameter	Type	Default	Description
`polarity`	`str`	server default (failures only)	`"failure"`, `"proper"` (healthy tally), or `"all"` for both
`status`	`str`	none	e.g. `"open"`, filter to one status
`severity`	`str`	none	`"low"`, `"medium"`, `"high"`, or `"critical"`
`agent_id`	`str`	none	Restrict to one agent (matches either the signal’s representative agent or its occurrence trail)
`limit`	`int`	`50`	Capped at 100 server-side

list()/get() both return a signal with .id, .type, .severity, .polarity, .status, .summary, .pattern_key, .occurrence_count, .occurrences, .recommended_actions, .root_cause, and more, matching the fields shown in the dashboard’s triage queue.

`client.monitor.profile`

Get/update one agent’s Monitor coverage settings: coverage mode, sample rate, retention, redaction, and approval policy. Legacy threshold_overrides fields still round-trip for wire compatibility, but latency is a KPI metric now (Overview’s p95 card, straight from the traces) rather than a detection threshold.

profile = client.monitor.profile.get("agent_123")
print(profile.coverage_mode if profile else "never configured, on defaults")

client.monitor.profile.update("agent_123", coverage_mode="all", sample_rate=0.5)

get() returns None when the agent has never been configured (still on platform defaults). update() upserts and only changes the fields you pass:

Parameter	Type	Description
`enabled`	`bool`	Turn Monitor on/off for this agent
`failure_detection_enabled` / `info_detection_enabled`	`bool`	Opt a whole detection category out
`coverage_mode`	`str`	`"all"` (every trace) or `"sampled"`
`sample_rate`	`float`	Fraction of traffic monitored when `coverage_mode="sampled"`
`channels`	`list[str]`	Notification channels
`dataset_id`	`str`	Evaluation dataset this agent’s signals feed into
`threshold_overrides`	`dict`	Per-check threshold overrides, e.g. `{"latencyMs": 15000}`
`retention_days`	`int`	How long monitored traces are kept
`redaction_mode`	`str`	`"none"`, `"standard"`, or `"strict"`
`approval_policy`	`dict[str, str]`	Per-action approval mode for autotune actions

Self-host extras

A self-hosted instance adds two Monitor surfaces the hosted platform doesn’t have yet, both reachable from this same SDK:

Online evaluators (client.monitor.online_evaluators) - a real LLM judge scoring a sample of live traffic continuously, per trace or per whole session (scope="session", judged automatically once the conversation goes idle). See Online evaluators.
Outcome reports and user feedback (client.outcomes.report(...) / client.feedback.report(...)) - record what actually happened after the fact (a reopened ticket, an end user’s thumbs-down) against a trace; negative feedback raises a signal directly, and both streams feed the dashboard’s Judge Calibration card. See Outcomes & Judge Calibration.

Tracing

Send the traces this feature monitors

Patterns

The condition builder, match targets, and where signals go

Live Traffic Scoring

Ground Truth & Calibration

Online Evaluation (Monitor)

Two ways to trigger it

Trace-time: `monitor` and `pattern_ids`

Dashboard toggle: automatic, every trace from an agent

What gets checked

Scorer administration as code

`client.monitor.patterns`

`builder()` parameters

Where signals show up

`client.monitor.signals`

`list()` parameters

`client.monitor.profile`

Self-host extras

Tracing

Patterns

​Two ways to trigger it

​Trace-time: monitor and pattern_ids

​Dashboard toggle: automatic, every trace from an agent

​What gets checked

​Scorer administration as code

​client.monitor.patterns

​builder() parameters

​Where signals show up

​client.monitor.signals

​list() parameters

​client.monitor.profile

​Self-host extras

Tracing

Patterns

Two ways to trigger it

Trace-time: `monitor` and `pattern_ids`

Dashboard toggle: automatic, every trace from an agent

What gets checked

Scorer administration as code

`client.monitor.patterns`

`builder()` parameters

Where signals show up

`client.monitor.signals`

`list()` parameters

`client.monitor.profile`

Self-host extras