> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Human Review Queue

> Label ordinary traffic, not just what got flagged, and measure the judge against it

Every judge calibration story starts the same way: you look at what the judges flagged, agree or
disagree, and tune. That loop has a hole in it. Reviewing only flagged traces can tell you the
judge cries wolf too often, but it can never tell you the judge is **quietly scoring bad answers
as good** - those traces never reach you, because nothing flagged them.

The review queue closes that hole. It holds traces that raised no signal at all, waiting for a
human verdict.

## Getting traces into it

Two ways, both visible in **Review → Review samples**:

<CardGroup cols={2}>
  <Card title="Send one by hand" icon="hand-pointer">
    Open any trace in Observe and click **Send to review**. Use it when something looks off and
    you want a second opinion on the record.
  </Card>

  <Card title="Sample it automatically" icon="filter">
    A [rule](/monitor/rules) with the **Send to review** action samples live traffic into the
    queue continuously - 10% of everything, or only errored traces, or only traces mentioning
    "refund".
  </Card>
</CardGroup>

The queue is capped at 200 pending items - a fixed cap, not configurable. Past that, new traces
are refused with a reason rather than silently dropped, because a queue nobody can finish is not
a backlog, it is a leak. Work it down or narrow the rule.

## Labeling

The verdict vocabulary here is deliberately different from the signals queue. There is no claim
to confirm or reject - nothing was flagged - only a judgment on the answer itself:

* **Good** or **Bad**, plus an optional note.
* When a judge already scored that trace, a **corrected score** box appears: what the rating
  should have been, 0-10.

That last pair is the point. A human "bad" next to a judge's 8.5 is exactly the evidence that a
judge is too generous, and it is the pair
[outcome calibration](/monitor/outcomes) consumes. The score box only appears when there is a
score to correct, because a corrected rating with nothing to compare it against is noise.

`j` / `k` move through the queue, `g` labels good, `b` labels bad.

## What it feeds

Labels flow into the same confusion matrix as production outcome reports, reported separately as
`reviewLabelCount`, so you can see judge agreement measured on ordinary traffic rather than only
on the traffic the judge itself chose to flag. A judge that looks precise on flagged traces and
disagrees constantly on sampled ones is a judge with a blind spot, and this is the only place
that shows up.

## From the Python SDK

The whole loop is scriptable as `client.monitor.review_queue` - queue a suspicious trace from
the same script that found it, or batch-label from a notebook:

```python theme={null}
item = client.monitor.review_queue.queue(trace_id, note="Looks off, second opinion please")

for pending in client.monitor.review_queue.list(status="pending"):
    print(pending.trace_id, pending.judge_score_at_queue)

client.monitor.review_queue.label(item.id, "bad", corrected_score=2, note="Invented a refund window")
client.monitor.review_queue.dismiss(item.id)  # remove without a verdict
```

`list()` also takes `source=` (`"manual"` | `"rule"` | `"all"`) and `limit=` (default 100);
`status` accepts `"pending"`, `"labeled"`, `"skipped"`, or `"all"`.
`label` takes `"good"` or `"bad"`; `corrected_score` (0-10) only makes sense when a judge
already scored the trace. Labels and corrected scores feed per-scorer
[calibration](/monitor/outcomes) and become evidence for **Tune judge**, exactly like labels
recorded in the dashboard.

## API

```bash theme={null}
# Queue a trace
curl -X POST "$AGENTX_URL/api/v1/agent-monitoring/review-queue" \
  -H "x-api-key: $AGENTX_API_KEY" -H "content-type: application/json" \
  -d '{"traceId": "<trace>"}'

# What is waiting
curl "$AGENTX_URL/api/v1/agent-monitoring/review-queue?status=pending" \
  -H "x-api-key: $AGENTX_API_KEY"

# Record a verdict
curl -X PATCH "$AGENTX_URL/api/v1/agent-monitoring/review-queue/<id>" \
  -H "x-api-key: $AGENTX_API_KEY" -H "content-type: application/json" \
  -d '{"label": "bad", "correctedScore": 3, "note": "Invented a refund window"}'
```

Refusals are explicit and carry their reason: an unknown trace is a 404, a trace already waiting
is a 409, and a full queue is a 429. None of them is a silent success.

Labeled items are stored and included in [backups](/self-host/backup) - a labeled corpus is real
human work, and losing it on a restore would mean doing it all again.