Online Evaluation (Monitor)

​

Doing it well

• Score the behaviors that hurt in production. LLM judge scorers run judge criteria (faithfulness, tool accuracy, goal completion) against sampled live traces and whole sessions; patterns catch the deterministic failures (errors, empty responses, PII, latency) at zero LLM cost.
• Tune the sample rate to your volume. Every judged trace is a real LLM call - start at sample_rate=1.0 while traffic is small and you’re validating criteria, then dial down as volume grows. Deterministic patterns stay at 100%: they’re free.
• Let users vote. End-user feedback (client.feedback.report) is the cheapest evaluation signal you’ll ever get - a downvote raises a signal directly, and real-world outcomes measure how often your judges agreed with reality.
• Feed findings back into offline tests. Low-scoring traces (and suspiciously perfect ones) become dataset cases - the online/offline loop is what makes both sides better.

• Monitor is the charts - health strip, healthy-rate and signal-volume trends, and per-scorer live judge score charts with their alert thresholds drawn in (click a card for the scored traces).
• Review is where a human passes verdicts on individual signals: a worst-first queue with the occurrence text and judge rationale as evidence, and Confirm / Fixed / Ignore / Wrong judgement verdict actions (Wrong judgement records the correction that feeds Judge Calibration and Tune judge), plus Add to dataset to turn the failure into an eval case. The classic filterable signal table lives under Review → Review signals.

​

Checking is automatic

• monitor=False opts that trace out of all six ingest-time passes - pattern/built-in detection, LLM judge scorers, scorer groups, custom/code/external scorers, automation rules, and topic classification. Use it for traffic that is already judged elsewhere, so a second pass doesn’t double the bill.
• source="eval-run" traces are skipped automatically, even without the flag: the run’s own evaluator already judges each case. An explicit monitor=True overrides the skip - that is someone deliberately pointing checks at eval traffic.
• monitor=True with pattern_ids restricts pattern detection to exactly those patterns: the built-in checks and each pattern’s own sample rate are bypassed, and only the named patterns run. Leaving monitor unset (or monitor=True without pattern_ids) keeps the full default sweep.

from agentx import AgentX

client = AgentX.from_env()

pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
    severity="high",
).publish()

with client.tracer.trace(
    "support-agent", monitor=True, pattern_ids=[pattern.id]
) as span:
    span.output = call_llm(query)

​

What gets checked

​

Scorer administration as code

# Templates are opt-in; enable()/disable() preserve what's already set and return the resulting list.
client.monitor.scorers.enable(["pii-in-response", "secrets-in-response"])
client.monitor.scorers.disable(["profanity-in-response"])
client.monitor.scorers.templates()          # keys, rules, enabled state

scorer = client.monitor.scorers.create_code(
    "Apology overload",
    "async def handler(input, output, expected, metadata, trace):\n"
    "    return 0.0 if str(output).lower().count('sorry') >= 2 else 1.0\n",
    language="python", sample_rate=0.5, alert_below=0.5,
)
client.monitor.scorers.dry_run(kind="code", language="python", script="...")  # nothing persisted
client.monitor.scorers.events(scorer["_id"], window="24h")  # per-check history
client.monitor.scorers.create_external("Policy endpoint", "https://scorer.internal/score")

scorer = client.monitor.judge_scorers.create(
    "Support quality",
    judge={"acceptanceCriteria": "Concrete, correct, cites the policy."},
    offline={"numberOfRequests": 2, "jaccardSimilarity": {"enabled": True}},
    online={"enabled": True, "sampleRate": 0.2, "alertThreshold": 6},
)
client.evaluations.run(dataset_id, subject, scorer_id=scorer.id)  # same rubric offline

# Or the builder, the unified successor of evaluations.settings.builder - rubric,
# offline profile, and live profile in one call:
scorer = client.monitor.judge_scorers.builder(
    "Support quality",
    acceptance_criteria="Concrete and correct.",
    tool_context="detailed",
    number_of_requests=2,
    live=True, sample_rate=0.25,
).publish()
client.monitor.judge_scorers.update(scorer.id, online={"enabled": False})  # sparse: rubric untouched
client.monitor.judge_scorers.update(scorer.id, online=None)                # detach the online profile
client.monitor.judge_scorers.calibration(scorer.id)   # tuning/ratings/events resolve the profile id

​

client.monitor.patterns

pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
).publish()

print(pattern.id)

client.monitor.patterns.get(pattern.id)   # fetch one
client.monitor.patterns.list()            # list every pattern in the workspace

​

builder() parameters

​

Where signals show up

​

client.monitor.signals

signals = client.monitor.signals.list(severity="high", limit=20)
for s in signals:
    print(s.id, s.severity, s.summary, s.occurrence_count)

signal = client.monitor.signals.get(signals[0].id)
print(signal.recommended_actions)

​

list() parameters

​

Reading production metrics

client.monitor.kpis(window="7d")       # Overview KPI strip: totalRuns, healthRate,
                                       # failureRate, downvoteRate, toolFailureRate,
                                       # p95LatencyMs, deltas vs the prior window
client.monitor.metrics(window="1d")    # Monitor metrics grid: bucketed spans by kind,
                                       # latency percentiles, tokens/cost, tool executions
                                       # and failures, platform attribution
client.monitor.calibration(window="7d")  # Judge Calibration: comparedCount, agreementRate,
                                         # falsePositiveRate, falseNegativeRate

​

metrics() filters

client.monitor.metrics(window="7d", agent="support-agent", status="error")
client.monitor.metrics(window="1d", framework="langchain")   # one platform's traffic

​

Human review and automation rules

• client.monitor.review_queue - list(), queue(trace_id), label(item_id, "good"|"bad", corrected_score=...), dismiss(item_id). Labels feed judge calibration and become judge-tuning evidence. See Review queue.
• client.monitor.rules - create(name, action, filter=..., sample_rate=..., action_config=...) where action is "review" (sample matches into the review queue), "dataset" (append matches as dataset cases), or "webhook" (POST the matching trace to your URL); plus list(), update(), delete(). action_config is required for "dataset" ({"datasetId": ...}) and "webhook" ({"url": ...}) - the engine returns 400 without it. See Rules.
• client.monitor.alert_rules - KPI alert rules: create(name, metric=, operator=, threshold=, window_minutes=, channels=[...], severity=, cooldown_minutes=, agent_id=) where metric is one of failureRate, toolFailureRate, p95LatencyMs, estimatedCostUsd, judgeFailures, traceCount, and channels are built by the slack(url), teams(url), pagerduty(key), email(address), webhook(url) helpers in agentx.monitor.alert_rules; plus list(), get(), update(), delete(), events(rule_id) (delivery history), test(rule_id) (a test page to every channel), preview(metric, window_minutes) (the live value), and run_sweep(). See Alert rules.

​

client.monitor.profile

profile = client.monitor.profile.get("agent_123")
print(profile.coverage_mode if profile else "never configured, on defaults")

client.monitor.profile.update("agent_123", coverage_mode="all", sample_rate=0.5)

​

Self-host extras

• Live LLM judge scoring - a real LLM judge scoring a sample of live traffic continuously, per trace or per whole session (scope="session", judged automatically once the conversation goes idle). Configure it as the online profile of a judge scorer (client.monitor.judge_scorers, or the legacy client.monitor.online_evaluators view). See LLM judge scorers.
• Scorer groups (client.monitor.scorer_groups) - scorers of any kind composed into one weighted 0-10 score with must-pass gates, usable as a dataset run’s grader and against live traffic, per trace or per whole session. Session helpers live on client.monitor.sessions (scores(), run_sweep(), spans(), coherence_check()). See Scorer groups.
• Outcome reports and user feedback (client.outcomes.report(...) / client.feedback.report(...)) - record what actually happened after the fact (a reopened ticket, an end user’s thumbs-down) against a trace; negative feedback raises a signal directly, and both streams feed the dashboard’s Judge Calibration card. See Outcomes & Judge Calibration.