Prompts
In the proposal dialog, after generating a suggestion: pick a dataset and Run validation. Every case runs twice - once with the current published prompt as the system prompt, once with the candidate (including any edits you made in the diff) - graded with that dataset’s own judge criteria. Multi-turn cases play out in full, each turn’s real reply threaded into the next.- The verdict compares pairwise-scored cases only (both variants got a rating), so one variant erroring can’t skew the averages.
- Per-case deltas are listed, regressions highlighted - a candidate that wins on average but breaks two specific cases shows exactly which two.
- Publishing appends the verdict to the version’s reasoning, so version history keeps the receipt.
Tool schemas
Same flow, different measurement: both definitions are given to the model for the same real queries, and each run is scored on whether the model chose the tool and formed valid arguments against the definition’s JSON schema: required arguments present, no unexpected keys, and each value checked against its property’stype, enum, and string pattern - so a “digits only” schema actually catches "#88231". Tool calls are simulated, never executed - what a tool definition controls is tool choice and argument shape, and that’s exactly what’s measured.
By default the queries come from the tool’s own production failure evidence - the exact requests the current definition mishandled - or pick a dataset instead.
An honest approximation
Validation runs dataset cases against (system prompt + a stock model), not your full agent - AgentX doesn’t own your code, the same boundary Model Portability documents. Baseline and candidate run under identical conditions, so the delta is the meaningful number even where absolute scores differ from your real agent’s. For the highest-fidelity signal, validate against cases curated from production: real inputs, real failure modes.From the API
{baselineAvg, candidateAvg, delta, regressions, verdict, summary} (verdict is improved / regressed / tie / insufficient). Optional: model (default gpt-4o-mini) and maxCases (default 12; cases with expected answers are prioritized).
The Improvement Inbox
Validation’s fully automatic form: a background sweep (every 10 minutes) watches every registered prompt and tool schema, and when one accumulates 3+ fresh failures in 24h, it generates the proposal and runs the validation on its own, then queues the result at the top of Improve → Suggestions - trigger reason, verdict chip, reasoning, and the full diff, with Publish and Dismiss as the only actions. The human’s job collapses to reading a measured verdict. Spend and noise controls: at most 2 new proposals per sweep, a 24-hour per-target cooldown (dismissing means “stop nagging”, not “retry in ten minutes”), never a second proposal while one is pending, and validation capped at 6 cases. Prompt validation uses the dataset from the most recent eval run tagged with that prompt’s name; with no tagged run, the proposal queues unvalidated rather than not at all.AGENTX_IMPROVEMENT_SWEEP=false disables the background sweep; the Check now button (or POST /evaluate/improve/inbox/sweep/run) triggers one on demand.
Prompt Management
Where prompt proposals come from
Tool Schemas
Where tool-definition proposals come from