Skip to main content
Every judge calibration story starts the same way: you look at what the judges flagged, agree or disagree, and tune. That loop has a hole in it. Reviewing only flagged traces can tell you the judge cries wolf too often, but it can never tell you the judge is quietly scoring bad answers as good - those traces never reach you, because nothing flagged them. The review queue closes that hole. It holds traces that raised no signal at all, waiting for a human verdict.

​
Getting traces into it

Two ways, both visible in Review → Review samples:

Send one by hand

Open any trace in Observe and click Send to review. Use it when something looks off and you want a second opinion on the record.

Sample it automatically

A rule with the Send to review action samples live traffic into the queue continuously - 10% of everything, or only errored traces, or only traces mentioning “refund”.
The queue is capped at 200 pending items - a fixed cap, not configurable. Past that, new traces are refused with a reason rather than silently dropped, because a queue nobody can finish is not a backlog, it is a leak. Work it down or narrow the rule.

​
Labeling

The verdict vocabulary here is deliberately different from the signals queue. There is no claim to confirm or reject - nothing was flagged - only a judgment on the answer itself:
  • Good or Bad, plus an optional note.
  • When a judge already scored that trace, a corrected score box appears: what the rating should have been, 0-10.
That last pair is the point. A human “bad” next to a judge’s 8.5 is exactly the evidence that a judge is too generous, and it is the pair outcome calibration consumes. The score box only appears when there is a score to correct, because a corrected rating with nothing to compare it against is noise. j / k move through the queue, g labels good, b labels bad.

​
What it feeds

Labels flow into the same confusion matrix as production outcome reports, reported separately as reviewLabelCount, so you can see judge agreement measured on ordinary traffic rather than only on the traffic the judge itself chose to flag. A judge that looks precise on flagged traces and disagrees constantly on sampled ones is a judge with a blind spot, and this is the only place that shows up.

​
From the Python SDK

The whole loop is scriptable as client.monitor.review_queue - queue a suspicious trace from the same script that found it, or batch-label from a notebook:
list() also takes source= ("manual" | "rule" | "all") and limit= (default 100); status accepts "pending", "labeled", "skipped", or "all". label takes "good" or "bad"; corrected_score (0-10) only makes sense when a judge already scored the trace. Labels and corrected scores feed per-scorer calibration and become evidence for Tune judge, exactly like labels recorded in the dashboard.

​
API

Refusals are explicit and carry their reason: an unknown trace is a 404, a trace already waiting is a 409, and a full queue is a 429. None of them is a silent success. Labeled items are stored and included in backups - a labeled corpus is real human work, and losing it on a restore would mean doing it all again.