> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Outcomes & Judge Calibration

> Report what actually happened, and measure the judges against it

Patterns, [online evaluators](/monitor/online-evaluators), and eval runs all score your agent with LLM judges - but did the judge get it right? Self-host lets you report what **actually happened** after the fact, then measures the judges against it:

```python theme={null}
client.outcomes.report(
    trace_id=trace_id,
    outcome="reopened",              # a free label in your own taxonomy
    is_negative=True,                # the polarity calibration compares against
    reason="Customer reopened the ticket within 3 days",
    reported_by="servicenow-webhook",
)
```

The intended caller is usually another system - an incident tracker's webhook, a CRM workflow - via `POST /outcomes` (`{"traceId", "outcome", "isNegative", "reason?", "reportedBy?"}`; `evaluationRunResultId` works in place of `traceId` for offline eval results). Overview's **Judge Calibration** card compares each report against whatever verdict AgentX recorded for the same trace at the time - pattern hits, online-evaluator scores - and shows the agreement rate: how often a negative outcome was flagged in advance (and how often a good one was wrongly flagged). That turns "trust the LLM judge" into a measured number, and tells you which evaluators are worth tightening.

Per-evaluator calibration is also readable from the SDK - agreement rate, misses,
over-flags, and the exact disagreement cases - and feeds straight into the scriptable
[judge-tuning loop](/monitor/online-evaluators#tuning-the-judge-from-the-sdk):

```python theme={null}
cal = client.monitor.online_evaluators.calibration(evaluator_id, window="7d")
print(f"{cal['agreements']}/{cal['withGroundTruth']} agreement, missed {cal['missed']}")
```

## End-user feedback

The other ground-truth stream is your own users' votes, forwarded with
`client.feedback.report(...)` - every vote (up and down) feeds the same calibration math as
outcome reports, so the judges get measured against real human reactions too. See
[User Feedback](/monitor/user-feedback) for the full story (trace chips, the triage signal,
the Downvote rate KPI).

Both streams are deliberately kept out of the "AgentX flagged it in advance" side of
calibration: they are the reports being calibrated against, never predictions that inflate
agreement.

## Tuning the judges themselves

Calibration doesn't just measure - it feeds back. Each online evaluator has its own agreement rate against recorded reality, split by direction: **missed** (judge passed it, reality said bad - criteria too generous) and **over-flagged** (judge flagged it, reality said fine - criteria too strict). From the evaluator's row menu, **Tune judge** opens the disagreement cases - human re-scores from signal triage ranked first, since they carry rationales - and generates a rewrite of the evaluator's own grading criteria that encodes the principles behind them.

What makes this loop stronger than the prompt/tool ones: judging is exactly reproducible, so validation isn't an approximation. The candidate criteria **re-judge the very cases the current criteria got wrong**, plus a control set of cases they got right (the anti-overfit guard), and agreement with recorded ground truth is measured directly: "agrees on 3/5 cases the current criteria got wrong, preserves 4/5 they got right (net +2)". Publishing updates the evaluator's config with normal version history, so every tuning is reversible.