Skip to main content
Patterns, online evaluators, and eval runs all score your agent with LLM judges - but did the judge get it right? Self-host lets you report what actually happened after the fact, then measures the judges against it:
The intended caller is usually another system - an incident tracker’s webhook, a CRM workflow - via POST /outcomes ({"traceId", "outcome", "isNegative", "reason?", "reportedBy?"}; evaluationRunResultId works in place of traceId for offline eval results). Overview’s Judge Calibration card compares each report against whatever verdict AgentX recorded for the same trace at the time - pattern hits, online-evaluator scores - and shows the agreement rate: how often a negative outcome was flagged in advance (and how often a good one was wrongly flagged). That turns “trust the LLM judge” into a measured number, and tells you which evaluators are worth tightening. Per-evaluator calibration is also readable from the SDK - agreement rate, misses, over-flags, and the exact disagreement cases - and feeds straight into the scriptable judge-tuning loop:

​
End-user feedback

The other ground-truth stream is your own users’ votes, forwarded with client.feedback.report(...) - every vote (up and down) feeds the same calibration math as outcome reports, so the judges get measured against real human reactions too. See User Feedback for the full story (trace chips, the triage signal, the Downvote rate KPI). Both streams are deliberately kept out of the “AgentX flagged it in advance” side of calibration: they are the reports being calibrated against, never predictions that inflate agreement.

​
Tuning the judges themselves

Calibration doesn’t just measure - it feeds back. Each online evaluator has its own agreement rate against recorded reality, split by direction: missed (judge passed it, reality said bad - criteria too generous) and over-flagged (judge flagged it, reality said fine - criteria too strict). From the evaluator’s row menu, Tune judge opens the disagreement cases - human re-scores from signal triage ranked first, since they carry rationales - and generates a rewrite of the evaluator’s own grading criteria that encodes the principles behind them. What makes this loop stronger than the prompt/tool ones: judging is exactly reproducible, so validation isn’t an approximation. The candidate criteria re-judge the very cases the current criteria got wrong, plus a control set of cases they got right (the anti-overfit guard), and agreement with recorded ground truth is measured directly: “agrees on 3/5 cases the current criteria got wrong, preserves 4/5 they got right (net +2)”. Publishing updates the evaluator’s config with normal version history, so every tuning is reversible.