POST /outcomes ({"traceId", "outcome", "isNegative", "reason?", "reportedBy?"}; evaluationRunResultId works in place of traceId for offline eval results). Overview’s Judge calibration card compares each report against whatever verdict AgentX recorded for the same trace at the time - pattern hits, online-evaluator scores - and shows the agreement rate: how often a negative outcome was flagged in advance (and how often a good one was wrongly flagged). That turns “trust the LLM judge” into a measured number, and tells you which evaluators are worth tightening.
Per-evaluator calibration is also readable from the SDK - agreement rate, misses,
over-flags, and the exact disagreement cases - and feeds straight into the scriptable
judge-tuning loop:
alpha explained on the
online evaluators page) is one
call:
End-user feedback
The other ground-truth stream is your own users’ votes, forwarded withclient.feedback.report(...) - every vote (up and down) feeds the same calibration math as
outcome reports, so the judges get measured against real human reactions too. See
User Feedback for the full story (trace chips, the triage signal,
the Downvote rate KPI).
Both streams are deliberately kept out of the “AgentX flagged it in advance” side of
calibration: they are the reports being calibrated against, never predictions that inflate
agreement.
Human labels on unflagged traffic
Both streams above still depend on something reaching you. Neither can catch a judge quietly scoring bad answers as good, because those traces never surface. The human review queue is the third stream and the one that closes that gap: traces that raised no signal at all, labeled good or bad by a person, optionally with a corrected score next to what the judge gave. Those labels feed this same confusion matrix, counted separately asreviewLabelCount.
Tuning the judges themselves
Calibration doesn’t just measure - it feeds back. Each online evaluator has its own agreement rate against recorded reality, split by direction: missed (judge passed it, reality said bad - criteria too generous) and over-flagged (judge flagged it, reality said fine - criteria too strict). From the evaluator’s row menu, Tune judge opens the disagreement cases - when several ground-truth sources cover the same case, a human re-score from signal triage wins, since it carries a rationale - and generates a rewrite of the evaluator’s own grading criteria that encodes the principles behind them. The disagreement evidence draws on all three ground-truth streams: outcome reports, user feedback, and review-queue labels - a human “bad” with a corrected score next to the judge’s 8.5 is exactly the kind of case a rewrite gets built from. What makes this loop stronger than the prompt/tool ones: judging is exactly reproducible, so validation isn’t an approximation. The candidate criteria re-judge the very cases the current criteria got wrong, plus a control set of cases they got right (the anti-overfit guard), and agreement with recorded ground truth is measured directly: “agrees on 3/5 cases the current criteria got wrong, preserves 4/5 they got right (net +2)”. Publishing updates the evaluator’s config with normal version history, so every tuning is reversible. Publishing is also gated on that verdict: the publish call must carry the validation result (the SDK’spublish_tuning(..., validation=verdict), or validation in the request body), and
a measured regression is refused outright. force: true publishes anyway - a deliberate
escape hatch, not a default. The published version’s history entry records the provenance
either way: [judge tuning: validated improved, net agreement +2] for a validated publish,
[judge tuning: published without validation] for a forced one - so six months later you can
still tell which criteria changes were measured and which were vibes.