Skip to main content
Golden datasets shouldn’t only be hand-authored: the best regression tests are the exchanges your agent actually got wrong. AgentX turns any production trace or session into a dataset case, so every flagged failure can become a permanent test - the flywheel that keeps your eval suite honest about what production actually looks like.

​
Where to find it

  • Any trace: open a trace’s detail view (Observe → click a trace, or click through from a signal’s occurrence) → Add to dataset.
  • Any session: open a session’s detail view (Observe → Sessions) → Add as test case. The whole conversation becomes one multi-turn case: turn 1 is the main question, later turns become follow-up questions - a real failed conversation becomes a multi-turn regression test the runner already knows how to execute.

​
The flow

  1. Preview: AgentX extracts the user question(s) from what was actually sent - for traces carrying threaded conversation history, each turn’s question is that turn’s last user message, not the whole history.
  2. Edit: every question is editable, and each turn shows what the agent actually replied (or how it errored) for context. The expected answer is yours to write - Suggest drafts a corrected reference answer with one judge call, as a starting point, never a final answer.
  3. Pick a dataset and add. The case lands in the dataset’s questions with a normal version history entry.

​
Deduplication

Adding the same failure forty times helps nobody, so every add runs three escalating checks:
  1. Same source - this exact trace or session was already added to that dataset.
  2. Same question - normalized exact match against existing cases.
  3. Similar question - embedding similarity against existing cases (needs OPENAI_API_KEY; silently skipped without one). Paraphrases (“What is your policy for refunds?” vs “Hi, what’s your refund policy?”) are caught; related-but-distinct questions (“How long does a refund take?”) are not.
A duplicate comes back as a warning with the existing case quoted - Add anyway overrides it deliberately.

​
Provenance

Cases born from production carry a source field (traceId/sessionId/signalId) and show a Prod badge in the dataset editor, so you always know which tests came from reality versus imagination. Provenance survives later dataset edits.

​
From the API

The same three-step contract is scriptable (same routes on both the dashboard’s /evaluate mount and the SDK-facing /custom-agent-evaluations mount):

Sessions

The conversation-level surface these cases come from

Validating Proposals

What production-born cases are for: measuring candidate fixes

​
Synthetic cases from a source document

The other direction - no production traffic yet - is covered by AI-Assisted Dataset Generation, the dashed panel at the top of the Create/Edit Dataset dialog: paste the source your agent should be graded against (policy text, API docs, an FAQ) and the engine generates grounded test cases - varied phrasing and difficulty, expected results drawn only from the source, and at least one query the source deliberately does not answer (the correct behavior being to say so). Review the checklist and add the keepers; nothing lands in the dataset unreviewed. Kept cases are appended to the dialog’s question list - you still review and save the dataset normally. Input format: pasted plain text or Markdown only. There is no document parser - no PDF, Word, or HTML extraction. For a PDF or Word document, copy the text out and paste it; for a web page, paste the readable text rather than the page source (HTML tags are treated as literal text, which wastes the generator’s attention). The API takes the same shape: POST /evaluate/synthesize-cases with { sourceText, count, guidance?, datasetId? }, where sourceText is the raw text and datasetId optionally few-shots the style of an existing dataset’s cases.