Online Evaluation (Monitor)

Online evaluation scores traffic as it happens: every checked trace is evaluated the moment it arrives, with no dataset and no separate run. Unlike offline evaluation, there is no ground truth here - nobody wrote an expected answer for a live user’s question - so verdicts come from LLM-as-judge scoring and deterministic detectors running against real production behavior, exactly where your users experience it. In the dashboard this is the Monitor tab. When it finds something, the finding loops back: a bad trace becomes an offline dataset case in one click, so today’s production failure is tomorrow’s regression test. Doing it well:

Score the behaviors that hurt in production. Online evaluators run judge criteria (faithfulness, tool accuracy, goal completion) against sampled live traces and whole sessions; patterns catch the deterministic failures (errors, empty responses, PII, latency) at zero LLM cost.
Tune the sample rate to your volume. Every judged trace is a real LLM call - start at sample_rate=1.0 while traffic is small and you’re validating criteria, then dial down as volume grows. Deterministic patterns stay at 100%: they’re free.
Let users vote. End-user feedback (client.feedback.report) is the cheapest evaluation signal you’ll ever get - a downvote raises a signal directly, and real-world outcomes measure how often your judges agreed with reality.
Feed findings back into offline tests. Low-scoring traces (and suspiciously perfect ones) become dataset cases - the online/offline loop is what makes both sides better.

Monitor is AgentX’s automatic quality monitoring for production traffic. Every checked trace is evaluated against a set of detectors, and a match becomes a signal: a deduped, triage-ready record readable straight from the SDK via client.monitor.signals. In the dashboard the machinery splits into two surfaces: Monitor is the charts - health strip, healthy-rate and signal-volume trends, and per-scorer live judge score charts with their alert thresholds drawn in (click a card for the scored traces) - while the new Review tab is where a human passes verdicts on individual signals: a worst-first queue with the occurrence text and judge rationale as evidence, and Confirm / False positive / Dismiss / Add-to-dataset actions (False positive feeds Judge Calibration and Tune judge). The classic filterable signal table lives under Review > All signals. A pattern is a detection rule with a real id, the same way a dataset or an evaluation config has one. Build a pattern once with client.monitor.patterns.builder(...).publish(), then reference it by id, or rely on the built-in checks and any patterns your workspace has defined in the dashboard. This works the same way for an agent built natively in AgentX and for an external agent traced entirely through the Python SDK.

Two ways to trigger it

They can be combined; neither requires changes to how you already call tracer.trace(...) beyond the flags described below.

Trace-time: `monitor` and `pattern_ids`

Pass monitor=True on tracer.trace(...) to check that specific trace immediately, with no dashboard setup at all. pattern_ids restricts detection to exactly those patterns; omit it to run the full default sweep (built-in checks plus every active pattern) instead. monitor=False is the opposite: it opts that trace out of every ingest-time check - patterns, online evaluators, topic classification. Use it for traces produced inside evaluation runs, where the dataset’s own judge already scores each case and a second judging pass would only double the bill. Leaving monitor unset keeps the engine’s normal behavior.

from agentx import AgentX

client = AgentX.from_env()

pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
    severity="high",
).publish()

with client.tracer.trace(
    "support-agent", monitor=True, pattern_ids=[pattern.id]
) as span:
    span.output = call_llm(query)

pattern_ids fully defines what’s checked when provided: only those named patterns run, the built-in checks are skipped. This mirrors how scorer_id fully defines an evaluation’s grading config rather than layering on top of a default. Omit pattern_ids (keep just monitor=True) to run the full default sweep instead.

Dashboard toggle: automatic, every trace from an agent

Enable monitoring once per agent in the dashboard, and every subsequent trace from that agent is checked automatically, with no monitor=True needed on any individual call.

Send at least one trace. The first call to tracer.trace(...) for a given agent name auto-creates a reference agent in your workspace. See Tracing if you haven’t wired this up yet.
Open Governance > Agents. Your SDK-traced agent appears in the agent list with an External badge, alongside any native agents.
That’s it - monitoring is on. Every ingested trace is checked by the scorers you enable. There is no global sample rate: sampling lives on the scorers that spend LLM budget (each judge scorer’s own rate, the Topics classification rate), so free deterministic checks always see all traffic.

What gets checked

Three separate streams classify a monitored trace: Operational outcomes - facts the trace itself recorded, always on, never configurable, and never raising triage signals (they feed the KPI failure metrics and Top failing instead):

Operational outcome	Recorded when
Tool/action failure	A recorded tool call reports `success: false`
Trace error	`error` was set on the trace (e.g. via `span.set_error(...)`, or an exception inside a decorated function)
Empty response	The traced output is empty

Latency is a distribution metric (Overview’s p95 card, straight from the traces), not a failure classification. Scorers - judgments you opt in to from the Scorers page, all off by default. Six built-in template scorers ship with the engine, all zero-LLM-cost; your own template patterns, LLM judge scorers, and custom endpoint scorers layer on top:

Built-in template scorer	Flags when the response contains	Severity
Secrets in response	A leaked credential: API keys (OpenAI/AWS/GitHub/Slack), bearer tokens, JWTs, private-key blocks	critical
PII in response	Personal data: emails, phone numbers, SSNs, payment cards	high
Prompt injection echo	A known jailbreak phrasing repeated in the output (“ignore previous instructions”, system-prompt disclosure)	high
Profanity in response	Profanity from a small unambiguous wordlist	medium
Refusal / non-answer	A deflection (“I can’t help with that”) - sometimes correct behavior, surfaced for triage	low
Malformed JSON response	Output that isn’t parseable JSON - enable only for agents whose contract is JSON	medium

Human feedback - your users’ votes via client.feedback.report(trace_id, "down"), raising a Negative user feedback signal directly (see User Feedback).

Metric semantics: operational outcomes and template/pattern hits classify the RUN (they move failureRate and the run-outcome breakdown); judge, code, and external scorer verdicts are evaluator events - they raise signals and keep per-scorer histories without reclassifying the run. See what moves which metric.

Scorer administration as code

Everything the Scorers page does is scriptable via client.monitor.scorers - enable the shipped templates, deploy code/external scorers, and read their check history:

# Templates are opt-in; enable() preserves what's already on and returns the resulting list.
client.monitor.scorers.enable(["pii-in-response", "secrets-in-response"])
client.monitor.scorers.templates()          # keys, rules, enabled state

scorer = client.monitor.scorers.create_code(
    "Apology overload",
    "async def handler(input, output, expected, metadata, trace):\n"
    "    return 0.0 if str(output).lower().count('sorry') >= 2 else 1.0\n",
    language="python", sample_rate=0.5, alert_below=0.5,
)
client.monitor.scorers.dry_run(kind="code", language="python", script="...")  # nothing persisted
client.monitor.scorers.events(scorer["_id"], window="24h")  # per-check history
client.monitor.scorers.create_external("Policy endpoint", "https://scorer.internal/score")

LLM judges have their own unified surface, client.monitor.judge_scorers: one scorer = one judge rubric + an offline profile (dataset-run grading; the scorer’s id is exactly the scorer_id a run takes - evaluation_settings_id remains a working alias) + an optional online profile (live-traffic scoring). Strictly one online profile per scorer.

scorer = client.monitor.judge_scorers.create(
    "Support quality",
    judge={"acceptanceCriteria": "Concrete, correct, cites the policy."},
    offline={"numberOfRequests": 2, "jaccardSimilarity": {"enabled": True}},
    online={"enabled": True, "sampleRate": 0.2, "alertThreshold": 6},
)
client.evaluations.run(dataset_id, subject, scorer_id=scorer.id)  # same rubric offline

# Or the builder, the unified successor of evaluations.settings.builder - rubric,
# offline profile, and live profile in one call:
scorer = client.monitor.judge_scorers.builder(
    "Support quality",
    acceptance_criteria="Concrete and correct.",
    tool_context="detailed",
    number_of_requests=2,
    live=True, sample_rate=0.25,
).publish()
client.monitor.judge_scorers.update(scorer.id, online={"enabled": False})  # sparse: rubric untouched
client.monitor.judge_scorers.update(scorer.id, online=None)                # detach the online profile
client.monitor.judge_scorers.calibration(scorer.id)   # tuning/ratings/events resolve the profile id

The older per-profile surfaces (client.evaluations.settings, client.monitor.online_evaluators) keep working unchanged as views onto the same entity - each emits a default-hidden DeprecationWarning on first use pointing here. Projects and trace read-back are first-class too: client.projects.create(name) returns an isolated project with its own apiKey (the per-run isolation pattern integration suites use), and client.traces.get(trace_id) / client.traces.list(cursor=...) read what the tracer wrote. Pattern scorers you author are rules matched against the response text or the trace: keyword/contains, regex, or an LLM-judged semantic rubric. Build them via the dashboard (Scorers → New scorer → Pattern scorer - see Patterns for the condition builder, AND/OR/NOR combination, and match targets) or with client.monitor.patterns.builder(...), shown below.

The “Tool/action failure” check reads the success: false a recorded tool call carries. The easiest way to produce that is tracer.trace_tool_call(...), which captures an escaping exception as a failed call automatically - see Tracing.

A trace that matches nothing becomes a healthy “info” tally instead, which powers the agent’s health-rate percentage.

`client.monitor.patterns`

pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
).publish()

print(pattern.id)

client.monitor.patterns.get(pattern.id)   # fetch one
client.monitor.patterns.list()            # list every pattern in the workspace

`builder()` parameters

Parameter	Type	Default	Description
`name`	`str`	required	Pattern display name
`description`	`str`	none	Human-readable description
`detector_kind`	`str`	`"contains"`	`"contains"`, `"regex"`, or `"semantic"`, selects which field below is used
`match_target`	`list[str]`	`["response"]`	Where to look: `"response"`, `"userMessage"`, `"trace"`
`match_mode`	`str`	`"any"`	For `detector_kind="contains"`: `"any"` or `"all"` of `include_terms` must match
`include_terms` / `exclude_terms`	`list[str]`	`[]`	Phrases to require / exclude, for `detector_kind="contains"`
`regex`	`str`	none	Regular expression body, for `detector_kind="regex"`
`semantic_prompt`	`str`	none	Rubric an LLM judges the response against, for `detector_kind="semantic"`
`severity`	`str`	`"medium"`	`"low"`, `"medium"`, `"high"`, or `"critical"`
`polarity`	`str`	`"failure"`	`"failure"` raises a signal to triage; `"proper"` logs a healthy tally instead
`enabled`	`bool`	`True`	Whether the pattern is checked at all
`sample_rate`	`float`	`1.0`	Fraction of matching traces to actually check, `0.0` to `1.0`
`scope_mode` / `agent_ids`	`str` / `list[str]`	`"all"` / `[]`	Restrict this pattern to specific agents instead of the whole workspace

publish() returns a pattern with .id, which you pass in pattern_ids at trace time.

On the hosted platform, creating a pattern requires a Business or Enterprise plan (the same entitlement gate as the dashboard’s Patterns UI). Self-host has no plan gates.

Where signals show up

Matches are deduped by pattern and (usually) by agent, so a recurring issue accumulates occurrences on one signal instead of creating a new row every time. Each signal links back to the trace and tool calls that produced it, so a reviewer can open the exact exchange from the dashboard (Governance > Observe), or read it straight from the SDK.

`client.monitor.signals`

Read-only: a signal is the system’s output from checking traces against patterns, not something you create directly.

signals = client.monitor.signals.list(severity="high", limit=20)
for s in signals:
    print(s.id, s.severity, s.summary, s.occurrence_count)

signal = client.monitor.signals.get(signals[0].id)
print(signal.recommended_actions)

`list()` parameters

Parameter	Type	Default	Description
`polarity`	`str`	server default (failures only)	`"failure"`, `"proper"` (healthy tally), or `"all"` for both
`status`	`str`	none	e.g. `"open"`, filter to one status
`severity`	`str`	none	`"low"`, `"medium"`, `"high"`, or `"critical"`
`agent_id`	`str`	none	Restrict to one agent (matches either the signal’s representative agent or its occurrence trail)
`limit`	`int`	`50`	Capped at 100 server-side

list()/get() both return a signal with .id, .type, .severity, .polarity, .status, .summary, .pattern_key, .occurrence_count, .occurrences, .recommended_actions, .root_cause, and more, matching the fields shown in the dashboard’s triage queue.

Reading production metrics

The numbers behind the dashboard’s Overview and Monitor charts are readable as plain dicts, for wiring into your own dashboards and alerts:

client.monitor.kpis(window="7d")       # Overview KPI strip: totalRuns, healthRate,
                                       # failureRate, downvoteRate, toolFailureRate,
                                       # p95LatencyMs, deltas vs the prior window
client.monitor.metrics(window="1d")    # Monitor metrics grid: bucketed spans by kind,
                                       # latency percentiles, tokens/cost, tool executions
                                       # and failures, platform attribution
client.monitor.calibration(window="7d")  # Judge Calibration: comparedCount, agreementRate,
                                         # falsePositiveRate, falseNegativeRate

kpis() and calibration() take a window of "24h", "7d" (default), or "30d".

`metrics()` filters

metrics() accepts a window from "1h" to "90d" (default "1d") plus optional filters that scope every number the way the dashboard’s filter chips do:

client.monitor.metrics(window="7d", agent="support-agent", status="error")
client.monitor.metrics(window="1d", framework="langchain")   # one platform's traffic

Filter	Description
`agent`	Only traffic from this agent
`model`	Only spans that ran on this model
`tool`	Only executions of this tool
`framework`	The platform label traces carry (see Platform detection); `"other"` selects unlabeled traffic
`status`	Run status, e.g. `"error"`

The response includes platform attribution - frameworks window totals plus per-bucket byFramework - which is the data behind the Platforms chart. Two related project-wide switches: client.monitor.topics(window) reads the Topics view (LLM-classified themes of sampled traffic - empty until enabled), and client.monitor.set_topics(True, sample_rate=0.2) turns classification on (each classified trace costs one judge call, hence the sample rate).

Human review and automation rules

The review loop and the routing rules the dashboard offers are scriptable too:

client.monitor.review_queue - list(), queue(trace_id), label(item_id, "good"|"bad", corrected_score=...), dismiss(item_id). Labels feed judge calibration and become judge-tuning evidence. See Review queue.
client.monitor.rules - create(name, action, filter=..., sample_rate=...) where action is "review" (sample matches into the review queue), "dataset" (append matches as dataset cases), or "webhook" (POST the matching trace to your URL); plus list(), update(), delete(). See Rules.

`client.monitor.profile`

Get/update one agent’s monitoring profile: enable/disable, detection category opt-outs, and approval policy. The coverage_mode/sample_rate pair still round-trips for wire compatibility but gates nothing - detection runs on all ingested traffic, and sampling lives on each scorer’s own rate. Legacy threshold_overrides fields also still round-trip; latency is a KPI metric now (Overview’s p95 card, straight from the traces) rather than a detection threshold.

profile = client.monitor.profile.get("agent_123")
print(profile.coverage_mode if profile else "never configured, on defaults")

client.monitor.profile.update("agent_123", coverage_mode="all", sample_rate=0.5)

get() returns None when the agent has never been configured (still on platform defaults). update() upserts and only changes the fields you pass:

Parameter	Type	Description
`enabled`	`bool`	Turn Monitor on/off for this agent
`failure_detection_enabled` / `info_detection_enabled`	`bool`	Opt a whole detection category out
`coverage_mode`	`str`	Legacy, gates nothing - detection always sees every trace
`sample_rate`	`float`	Legacy, gates nothing - use each scorer’s own `sample_rate`
`channels`	`list[str]`	Notification channels
`dataset_id`	`str`	Evaluation dataset this agent’s signals feed into
`threshold_overrides`	`dict`	Per-check threshold overrides, e.g. `{"latencyMs": 15000}`
`retention_days`	`int`	How long monitored traces are kept
`approval_policy`	`dict[str, str]`	Per-action approval mode for autotune actions

Self-host extras

A self-hosted instance adds three Monitor surfaces the hosted platform doesn’t have yet, both reachable from this same SDK:

Live LLM judge scoring - a real LLM judge scoring a sample of live traffic continuously, per trace or per whole session (scope="session", judged automatically once the conversation goes idle). Configure it as the online profile of a judge scorer (client.monitor.judge_scorers, or the legacy client.monitor.online_evaluators view). See Online evaluators.
Scorer groups (client.monitor.scorer_groups) - scorers of any kind composed into one weighted 0-10 score with must-pass gates, usable as a dataset run’s grader and against live traffic, per trace or per whole session. Session helpers live on client.monitor.sessions (scores(), run_sweep(), spans(), coherence_check()). See Scorer groups.
Outcome reports and user feedback (client.outcomes.report(...) / client.feedback.report(...)) - record what actually happened after the fact (a reopened ticket, an end user’s thumbs-down) against a trace; negative feedback raises a signal directly, and both streams feed the dashboard’s Judge Calibration card. See Outcomes & Judge Calibration.

Tracing

Send the traces this feature monitors

Patterns

The condition builder, match targets, and where signals go

​Two ways to trigger it

​Trace-time: monitor and pattern_ids

​Dashboard toggle: automatic, every trace from an agent

​What gets checked

​Scorer administration as code

​client.monitor.patterns

​builder() parameters

​Where signals show up

​client.monitor.signals

​list() parameters

​Reading production metrics

​metrics() filters

​Human review and automation rules

​client.monitor.profile

​Self-host extras

Tracing

Patterns

Two ways to trigger it

Trace-time: `monitor` and `pattern_ids`

Dashboard toggle: automatic, every trace from an agent

What gets checked

Scorer administration as code

`client.monitor.patterns`

`builder()` parameters

Where signals show up

`client.monitor.signals`

`list()` parameters

Reading production metrics

`metrics()` filters

Human review and automation rules

`client.monitor.profile`

Self-host extras