Skip to main content
Online evaluation exists to catch your agent failing in production. Auto-improve is what happens after the catch: the failures a human confirmed are spent - as one body of evidence - on an improvement report, and the report is applied to your agent’s actual source code by a coding agent. The loop runs entirely on online evidence: live-traffic verdicts a person vouched for, never offline dataset runs.

​
1. Confirm failures (Review)

In Review → Review signals, passing a Confirm verdict on a signal - a low judge score or a matched failure pattern - automatically lands the confirmed occurrence in the pending batch (“Confirmed failures”). Confirm is the accumulation gesture; there is no second button, and declining is choosing Ignore. The evidence (input, output, judge rationale, score) is snapshotted at confirm time, so later trace pruning cannot hollow it out. Re-confirming the same verdict after a reopen never double-counts. What makes this evidence unusual: every item was flagged by a machine and confirmed by a person. It is the high-precision slice of production traffic - the opposite of auto-selected “worst examples”.

​
2. Generate a report (Insights → Auto-improve)

Insights → Auto-improve shows the pending batch and its count. Generate improvement report runs one LLM pass (explicit and billed - never implicit) that clusters the confirmed failures into issues, each with a title, description, recommendation, and its evidence inline. Generating seals the batch: the group becomes that report’s permanent source-case set (renamed with its seal time) and the pending accumulator is cleared - the next Confirm starts a fresh batch, and the next generate produces a new report from a new group. One report per group, one group per report. The report is stored under its own id, shown with a copy button. A mis-confirmed item can be pruned from the batch before generating with client.monitor.improvement_groups.remove_member(group_id, member_id). From the SDK:

​
3. Apply the fixes (the auto-improve skill)

The report id is the hand-off to the AgentX Agent Skill plugin. In your agent’s repo, tell your coding agent:
The skill fetches the report from the engine (GET /agent-monitoring/improvement-reports/:id), then triages each recommendation against your real source rather than applying it literally - the recommendations were written by a judge that never saw the code, so they are hypotheses, not instructions. Each issue gets a verdict (apply / already handled / reject with a reason), and the surviving fixes are made in place.

​
Verifying the fix

The same scorers keep watching production. A fix that holds stops re-raising its signals; a signal you resolved as Fixed that fires again reopens as a regression - the engine telling you the fix did not hold. For a pre-deploy check, curate the confirmed failures into a dataset and run an offline evaluation against it.
Auto-improve is self-host only, and the report generator uses the engine’s default judge model (override with generate_report(group_id, model="...") or the route’s model field).