> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Validating Proposals

> Every proposed prompt or tool-schema rewrite ships with a measured verdict before a human publishes it

[Prompt Management](/improve/prompt-management) and [Tools](/improve/tool-schemas) generate improvement proposals from real evidence - but a plausible diff isn't proof. **Validate before publishing** runs the candidate and the current published version against the same real cases, under identical conditions, and shows the measured comparison. The human approving a rewrite approves a number ("candidate scored 7.8 vs 6.1 baseline on 18 cases, 2 regressions"), not a hunch.

## Prompts

In the proposal dialog, after generating a suggestion: pick a dataset and **Run validation**. Every case runs twice - once with the current published prompt as the system prompt, once with the candidate (including any edits you made in the diff) - graded with that dataset's own judge criteria. Multi-turn cases play out in full, each turn's real reply threaded into the next.

* The verdict compares **pairwise-scored** cases only (both variants got a rating), so one variant erroring can't skew the averages.
* Per-case deltas are listed, regressions highlighted - a candidate that wins on average but breaks two specific cases shows exactly which two.
* Publishing appends the verdict to the version's reasoning, so version history keeps the receipt.

## Tool schemas

Same flow, different measurement: both definitions are given to the model for the same real queries, and each run is scored on whether the model **chose the tool** and **formed valid arguments** against the definition's JSON schema: required arguments present, no unexpected keys, and each value checked against its property's `type`, `enum`, and string `pattern` - so a "digits only" schema actually catches `"#88231"`. Tool calls are simulated, never executed - what a tool *definition* controls is tool choice and argument shape, and that's exactly what's measured.

By default the queries come from the tool's own **production failure evidence** - the exact requests the current definition mishandled - or pick a dataset instead.

## An honest approximation

Validation runs dataset cases against (system prompt + a stock model), not your full agent - AgentX doesn't own your code, the same boundary [Model Portability](/evaluation/model-portability) documents. Baseline and candidate run under identical conditions, so the **delta** is the meaningful number even where absolute scores differ from your real agent's. For the highest-fidelity signal, validate against cases [curated from production](/evaluation/datasets-from-production): real inputs, real failure modes.

## From the API

```bash theme={null}
curl -X POST http://localhost:4700/api/v1/evaluate/prompts/<promptId>/proposals/validate \
  -H "x-api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"candidateText": "You are a meticulous support agent...", "datasetId": "<datasetId>"}'

curl -X POST http://localhost:4700/api/v1/evaluate/tool-schemas/<toolSchemaId>/proposals/validate \
  -H "x-api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"candidateDefinition": "{\"name\": \"search_orders\", ...}"}'
```

Both return the per-case pairs plus `{baselineAvg, candidateAvg, delta, regressions, verdict, summary}` (`verdict` is `improved` / `regressed` / `tie` / `insufficient`). Optional: `model` (default `gpt-4o-mini`) and `maxCases` (default 12; cases with expected answers are prioritized).

## The Improvement Inbox

Validation's fully automatic form: a background sweep (every 10 minutes) watches every registered prompt and tool schema, and when one accumulates **3+ fresh failures in 24h**, it generates the proposal *and* runs the validation on its own, then queues the result at the top of **Improve → Suggestions** - trigger reason, verdict chip, reasoning, and the full diff, with **Publish** and **Dismiss** as the only actions. The human's job collapses to reading a measured verdict.

Spend and noise controls: at most 2 new proposals per sweep, a 24-hour per-target cooldown (dismissing means "stop nagging", not "retry in ten minutes"), never a second proposal while one is pending, and validation capped at 6 cases. Prompt validation uses the dataset from the most recent eval run tagged with that prompt's name; with no tagged run, the proposal queues unvalidated rather than not at all. `AGENTX_IMPROVEMENT_SWEEP=false` disables the background sweep; the **Check now** button (or `POST /evaluate/improve/inbox/sweep/run`) triggers one on demand.

<CardGroup cols={2}>
  <Card title="Prompt Management" icon="pen-to-square" href="/improve/prompt-management">
    Where prompt proposals come from
  </Card>

  <Card title="Tool Schemas" icon="wrench" href="/improve/tool-schemas">
    Where tool-definition proposals come from
  </Card>
</CardGroup>