Head-to-Head Comparison

Two runs of the same dataset score 7.4 and 7.6. Did the change help? Honestly: you cannot tell. LLM judge scores drift when the rubric wording moves, and they bunch up in the 7-8 band, so a 0.2 gap between two averages is inside the noise. This is the reason teams stare at a green delta and still ship a regression. Head-to-head judging asks a different question, the one a person would actually ask: for this question, which of these two answers is better? A preference between two concrete answers is far more stable than either answer’s absolute score, and it is directly the claim you want to make when shipping a change.

Run one

comparison = client.evaluations.compare_pairwise(
    candidate_run_id,          # run A - the new version
    baseline_run_id,           # run B - what it has to beat
    both_orders=True,          # judge each pair twice, sides swapped
)

print(comparison.summary.winner)   # "a", "b", or "tie"
print(comparison.summary.a_wins, comparison.summary.b_wins, comparison.summary.ties)
print(comparison.summary.flip_rate)

curl -X POST "$AGENTX_URL/api/v1/evaluate/runs/pairwise" \
  -H "x-api-key: $AGENTX_API_KEY" \
  -H "content-type: application/json" \
  -d '{"runAId": "<candidate>", "runBId": "<baseline>", "bothOrders": true}'

Both runs must be of the same dataset - a per-question comparison needs the same questions. Run A is the candidate by convention, so “A wins” reads as “the new version won”. In the dashboard, the same thing lives under Head to head in the version comparison dialog, directly beneath the two averages it exists to second-guess. Nothing judges on open: the panel shows the last comparison for that pair, and judging again is a click, because every case costs a judge call.

Position bias, and why `both_orders` matters

LLM judges favor whichever answer they read first. Left alone, that bias decides comparisons, and you would never see it - the result looks like a clean verdict. AgentX designs against it rather than mentioning it:

The judge is shown “Answer 1” and “Answer 2”. It never learns which run is the candidate.
The presentation order alternates case by case, so a biased judge cannot favor one run across the whole batch.
both_orders=True judges every pair twice with the sides swapped. Any disagreement between the two passes is recorded as a tie, but only opposite verdicts (A in one order, B in the other) count as a position flip and raise the batch’s flip rate - the order decided it, not the answers. A tie-vs-win disagreement still becomes a tie, but it is ordinary judge wobble, not evidence of position bias, so it does not raise the flip rate.

A high flip rate does not mean the comparison narrowly passed. It means the comparison is inconclusive, and both the dashboard and assert_pairwise say so rather than reporting the win they would otherwise show. both_orders doubles the judge cost. That is the price of knowing whether the result is real.

Fail a build on it

from agentx.testing import assert_pairwise

def test_new_prompt_is_actually_better():
    comparison = client.evaluations.compare_pairwise(
        candidate_run_id, baseline_run_id, both_orders=True
    )
    assert_pairwise(
        comparison,
        must_win=True,        # a tie fails: "no worse" is not the claim being made
        max_losses=2,         # ...and it must not have broken more than 2 cases winning
        max_flip_rate=0.2,    # ...on a comparison that is not just position bias
    )

Each check catches something a single average cannot:

Check	Catches
`must_win`	A change that cannot beat what it replaces. A tie is a failure, not a pass.
`max_losses`	A change that lifts the average by improving easy cases while breaking hard ones.
`max_flip_rate`	A comparison decided by presentation order, which should never read as a pass.

A comparison run without both_orders has no flip rate to check, so max_flip_rate is skipped rather than passing on a fabricated zero.

What is graded

The head-to-head uses the same material a normal run of that dataset uses: its evaluation criteria and each case’s expected result as the reference answer. This matters more than it sounds. Without the reference, a confidently vague answer (“Maybe, it depends”) can beat a correct specific one, because the judge has no ground truth to check against. Override the criteria per comparison when you want to compare on one axis:

client.evaluations.compare_pairwise(
    candidate_run_id, baseline_run_id,
    criteria="Which answer is more concise while still covering the policy?",
)

Reading the result

for case in comparison.cases:
    print(case.query, case.winner, case.presented_first, case.justification)

Each case records which side the judge read first, so any verdict can be audited after the fact. comparison.summary.errors counts cases where the judge call itself failed (provider outage, unusable output) - those are excluded from the win/tie totals, never counted as real ties. Cases the two runs do not share - one side produced no answer, or the batch hit the 100-case per-comparison cap - are listed in comparison.skipped with a reason, rather than quietly dropped in a way that would make a partial comparison look like a sweep. Past comparisons are stored and included in backups:

client.evaluations.list_pairwise(run_a_id=candidate_run_id)
client.evaluations.get_pairwise(batch_id)

When to use which

Absolute scores

“Is this good enough to ship?” Use a rating floor - CI gates and min_rating. A threshold needs a number.

Head to head

“Is this better than what we have?” Use pairwise. A comparison needs a preference, and preferences survive judge drift that absolute numbers do not.

Most teams want both: a floor that stops bad releases, and a head-to-head that proves the change was worth shipping.

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Playground

Framework Examples

Head-to-Head Comparison

Run one

Position bias, and why `both_orders` matters

Fail a build on it

What is graded

Reading the result

When to use which

Absolute scores

Head to head

​Run one

​Position bias, and why both_orders matters

​Fail a build on it

​What is graded

​Reading the result

​When to use which

Absolute scores

Head to head

Run one

Position bias, and why `both_orders` matters

Fail a build on it

What is graded

Reading the result

When to use which