> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Head-to-Head Comparison

> Ask which run answered better, instead of trusting two averages a tenth of a point apart

Two runs of the same dataset score 7.4 and 7.6. Did the change help?

Honestly: you cannot tell. LLM judge scores drift when the rubric wording moves, and they bunch
up in the 7-8 band, so a 0.2 gap between two averages is inside the noise. This is the reason
teams stare at a green delta and still ship a regression.

Head-to-head judging asks a different question, the one a person would actually ask: **for this
question, which of these two answers is better?** A preference between two concrete answers is
far more stable than either answer's absolute score, and it is directly the claim you want to
make when shipping a change.

## Run one

<CodeGroup>
  ```python SDK theme={null}
  comparison = client.evaluations.compare_pairwise(
      candidate_run_id,          # run A - the new version
      baseline_run_id,           # run B - what it has to beat
      both_orders=True,          # judge each pair twice, sides swapped
  )

  print(comparison.summary.winner)   # "a", "b", or "tie"
  print(comparison.summary.a_wins, comparison.summary.b_wins, comparison.summary.ties)
  print(comparison.summary.flip_rate)
  ```

  ```bash REST theme={null}
  curl -X POST "$AGENTX_URL/api/v1/evaluate/runs/pairwise" \
    -H "x-api-key: $AGENTX_API_KEY" \
    -H "content-type: application/json" \
    -d '{"runAId": "<candidate>", "runBId": "<baseline>", "bothOrders": true}'
  ```
</CodeGroup>

Both runs must be of the same dataset - a per-question comparison needs the same questions. Run A
is the candidate by convention, so "A wins" reads as "the new version won".

In the dashboard, the same thing lives under **Head to head** in the version comparison dialog,
directly beneath the two averages it exists to second-guess. Nothing judges on open: the panel
shows the last comparison for that pair, and judging again is a click, because every case costs a
judge call.

## Position bias, and why `both_orders` matters

LLM judges favor whichever answer they read first. Left alone, that bias decides comparisons, and
you would never see it - the result looks like a clean verdict.

AgentX designs against it rather than mentioning it:

* The judge is shown "Answer 1" and "Answer 2". It never learns which run is the candidate.
* The presentation order **alternates case by case**, so a biased judge cannot favor one run
  across the whole batch.
* `both_orders=True` judges every pair **twice with the sides swapped**. Any disagreement
  between the two passes is recorded as a **tie**, but only **opposite verdicts** (A in one
  order, B in the other) count as a position flip and raise the batch's **flip rate** - the
  order decided it, not the answers. A tie-vs-win disagreement still becomes a tie, but it is
  ordinary judge wobble, not evidence of position bias, so it does not raise the flip rate.

A high flip rate does not mean the comparison narrowly passed. It means the comparison is
inconclusive, and both the dashboard and `assert_pairwise` say so rather than reporting the win
they would otherwise show.

`both_orders` doubles the judge cost. That is the price of knowing whether the result is real.

## Fail a build on it

```python theme={null}
from agentx.testing import assert_pairwise

def test_new_prompt_is_actually_better():
    comparison = client.evaluations.compare_pairwise(
        candidate_run_id, baseline_run_id, both_orders=True
    )
    assert_pairwise(
        comparison,
        must_win=True,        # a tie fails: "no worse" is not the claim being made
        max_losses=2,         # ...and it must not have broken more than 2 cases winning
        max_flip_rate=0.2,    # ...on a comparison that is not just position bias
    )
```

Each check catches something a single average cannot:

| Check           | Catches                                                                           |
| --------------- | --------------------------------------------------------------------------------- |
| `must_win`      | A change that cannot beat what it replaces. A tie is a failure, not a pass.       |
| `max_losses`    | A change that lifts the average by improving easy cases while breaking hard ones. |
| `max_flip_rate` | A comparison decided by presentation order, which should never read as a pass.    |

A comparison run without `both_orders` has no flip rate to check, so `max_flip_rate` is skipped
rather than passing on a fabricated zero.

## What is graded

The head-to-head uses the same material a normal run of that dataset uses: its evaluation
criteria and each case's expected result as the reference answer. This matters more than it
sounds. Without the reference, a confidently vague answer ("Maybe, it depends") can beat a
correct specific one, because the judge has no ground truth to check against.

Override the criteria per comparison when you want to compare on one axis:

```python theme={null}
client.evaluations.compare_pairwise(
    candidate_run_id, baseline_run_id,
    criteria="Which answer is more concise while still covering the policy?",
)
```

## Reading the result

```python theme={null}
for case in comparison.cases:
    print(case.query, case.winner, case.presented_first, case.justification)
```

Each case records which side the judge read first, so any verdict can be audited after the fact.
`comparison.summary.errors` counts cases where the judge call itself failed (provider outage,
unusable output) - those are excluded from the win/tie totals, never counted as real ties.
Cases the two runs do not share - one side produced no answer, or the batch hit the 100-case
per-comparison cap - are listed in `comparison.skipped` with a reason, rather than quietly
dropped in a way that would make a partial comparison look like a sweep.

Past comparisons are stored and included in [backups](/self-host/backup):

```python theme={null}
client.evaluations.list_pairwise(run_a_id=candidate_run_id)
client.evaluations.get_pairwise(batch_id)
```

## When to use which

<CardGroup cols={2}>
  <Card title="Absolute scores" icon="gauge">
    "Is this good enough to ship?" Use a rating floor - [CI gates](/sdk/ci-cd) and
    `min_rating`. A threshold needs a number.
  </Card>

  <Card title="Head to head" icon="scale-balanced">
    "Is this better than what we have?" Use pairwise. A comparison needs a preference, and
    preferences survive judge drift that absolute numbers do not.
  </Card>
</CardGroup>

Most teams want both: a floor that stops bad releases, and a head-to-head that proves the change
was worth shipping.