Build Dataset

A dataset is a named set of test cases plus scoring configuration. Every evaluation run locks a snapshot of the dataset at creation time, so editing a dataset later never changes the scoring of past runs. There are two ways to build one: the fluent builder, or importing from a CSV.

Programmatic builder

dataset = (
    client.evaluations.datasets
    .builder(
        name="Support Agent v2",
        number_of_requests=3,
        acceptance_criteria="Accurate, concise, grounded in docs.",
        rejection_criteria="No hallucinated policies.",
    )
    .add_case(
        query="How do I reset my password?",
        expected_results="Explain the password reset process step by step.",
    )
    .add_case(
        query="What payment methods do you accept?",
        expected_results="List supported payment methods clearly.",
    )
    .publish()
)

print(dataset.id)

`builder()` parameters

Parameter	Type	Default	Description
`name`	`str`	required	Dataset display name
`description`	`str`	none	Human-readable description
`number_of_requests`	`int`	`1`	How many times each question is run per evaluation, for consistency testing
`acceptance_criteria`	`str`	none	What a good response looks like, included in the LLM scoring prompt
`rejection_criteria`	`str`	none	What a bad response looks like, included in the LLM scoring prompt
`evaluation_criteria`	`str`	none	Scoring rubric, included in the LLM scoring prompt
`vector_similarity`	`bool`	`False`	Enable cosine similarity scoring against `expected_results`
`jaccard_similarity`	`bool`	`False`	Enable Jaccard token-overlap scoring against `expected_results`
`bleu_score`	`bool`	`False`	Enable BLEU scoring against `expected_results`
`rouge_score`	`bool`	`False`	Enable ROUGE-L scoring against `expected_results`
`similarity_model`	`str`	provider default	Embedding model used for vector similarity, e.g. `"text-embedding-3-small"`
`sovereignty_models`	`list[str]`	none	Enables Sovereignty & Portability: models to compare on this dataset. Discover valid ids with `client.evaluations.list_models()`
`code_scorers`	`list[dict]`	none	Deterministic JS scorers run per result alongside the judge; each entry is `{"name", "code"}` plus optional `"enabled"` (default `True`). See Code scorers
`judge_prompt`	`str`	none	Override the LLM-as-judge’s raw prompt template. See Configuring the judge
`judge_model`	`str`	none	Override the LLM-as-judge’s model (any OpenAI or Anthropic id from `client.evaluations.list_models()`)

Repetition statistics

With number_of_requests above 1, the run reports caseStatistics: per case, the averageRating, minRating, maxRating, and ratingVariance across the repetitions (cases need at least two rated repetitions; smoke-test variants are excluded). A case swinging between 3 and 9 is a different problem from a case stuck at 6, and only the variance tells them apart. Connector-driven runs - the dashboard invoking your connected agent endpoint - honor the dataset’s numberOfRequests too, clamped to 1-10.

`add_case()` parameters

Parameter	Type	Description
`query`	`str`	required, the question or prompt sent to the agent
`expected_results`	`str`	Optional, enables vector and Jaccard similarity scoring for this case
`expected_capabilities`	`list[str]`	Optional. Capabilities/tools a correct response should demonstrate, shown to the judge
`expected_knowledge_base`	`list[str]`	Optional. Knowledge sources a correct response should draw on, shown to the judge
`expected_delegations`	`list[str]`	Optional. Sub-agents a correct run should delegate to, shown to the judge
`follow_up_questions`	`list[dict]`	Optional. Follow-up turns for this case, each a dict with the same keys as a main question
`judge_guideline`	`str`	Optional extra grading instructions for this specific case, layered on top of the dataset’s criteria fields
`smoke_test_count`	`int`	Optional, 1-10. Ask this question that many extra ways each run (LLM-paraphrased server-side), to catch agents that break on phrasing rather than substance. See Smoke testing below
`smoke_test_guidance`	`str`	Optional, only used when `smoke_test_count` is set. Free text steering what kind of variants get generated (tone, adversarial phrasing, different languages, …)
`expected_tools`	`list[str]`	Optional. The tool calls a correct run of this case should make - trajectory-matched against the result’s linked trace, reported as a pass/fail “Trajectory match” scorer row. See Expected trajectories below
`trajectory_match_mode`	`str`	`"strict"` (default), `"unordered"`, `"superset"`, or `"subset"` - see below
`expected_retrieval_context`	`str \| list[str]`	Optional. The chunk(s) a correct retriever should fetch for this query - compared against the actually retrieved context with token-level Jaccard similarity, reported as a “Context match (jaccard)” scorer row (0-1), no judge call. See Expected retrieval context below
`splits`	`list[str]`	Optional. Named subsets this case belongs to, e.g. `["smoke"]` - a run can then target just one split. See Splits below

publish() sends the dataset to AgentX and returns a Dataset object with .id, which you pass as dataset_id to client.evaluations.run().

Expected trajectories

Agentic cases can pin down how the answer should be produced, not just what it should say. Declare the expected tool calls; when a result links its trace (return {"output": ..., "trace_id": span.trace_id} from your agent function), the engine compares the trace’s actual tool-call sequence deterministically:

.add_case(
    query="I want a refund for order A-1001, it arrived two weeks ago.",
    expected_results="Confirms eligibility within the 30-day window.",
    expected_tools=["lookup_order", "refund_policy"],
    trajectory_match_mode="unordered",
)

Mode	Passes when
`strict`	Same calls, same order
`unordered`	Same calls, any order
`superset`	Every expected call present; extra calls allowed
`subset`	No unexpected calls; missing some is allowed

The verdict lands on each result as a named scorer row alongside the judge rating - and the LLM judge itself also sees the full trajectory in its prompt, so written criteria about tool behavior are scoreable too.

Expected retrieval context

RAG cases can pin what a correct retriever should have fetched. At scoring time the engine compares it against the actually retrieved context - the retrieval_context your agent function returned, else the linked trace’s retrieval spans - using token-level Jaccard similarity, and reports a Context match (jaccard) scorer row (0-1) on each result. It’s deterministic and costs no judge call, so it’s safe to run on every retriever, chunking, or embedding change:

.add_case(
    query="What is the refund window?",
    expected_retrieval_context=[
        "Refund policy: refunds are available within 30 days of delivery for unused items."
    ],
)

The same field is editable in the dashboard (open a dataset case, Expected retrieval context, blank line between chunks). Jaccard measures token overlap, not meaning - pair it with the RAG judge metrics for semantic verdicts.

Smoke testing

Every variant of a smoke-tested case is graded against the same expected_results as the original question. Both the paraphrase text and the count are generated entirely server-side, .execute() asks the extra variants and submits them automatically:

dataset = (
    client.evaluations.datasets
    .builder(name="Support Agent v2")
    .add_case(
        query="How do I reset my password?",
        expected_results="Explain the password reset process step by step.",
        smoke_test_count=3,
        smoke_test_guidance="try terse, frustrated, and non-native-speaker phrasing",
    )
    .publish()
)

Your agent function can check case.is_smoke_test_variant if it wants to handle variants differently, but this isn’t required, case.query already holds the text to ask either way. Smoke-test results are scored normally but reported as a separate robustness signal, excluded from report.average_rating so one hard phrasing doesn’t skew your headline score. Only a case’s opening question can be smoke-tested, smoke_test_count/smoke_test_guidance are ignored on follow-up questions.

Splits

One golden dataset usually serves two budgets: a handful of cases on every PR, everything on a schedule. Tag cases into named subsets and pick the subset at run time instead of maintaining two datasets:

dataset = (
    client.evaluations.datasets
    .builder(name="Support Agent v2")
    .add_case(query="How do I reset my password?", expected_results="...", splits=["smoke"])
    .add_case(query="What payment methods do you accept?", expected_results="...")
    .publish()
)

client.evaluations.run(dataset.id, subject, split="smoke").execute(my_agent).finalize()

A run with split="smoke" executes only the tagged cases; omit split to run everything. The dashboard’s Run dialog and connector-driven runs take the same split. Case indexes are preserved across split and full runs, so per-case history lines up whichever subset ran.

CSV import

from_csv() returns the same builder as above with one case per row, so you can keep chaining (.add_case(...)) and must finish with .publish():

dataset = client.evaluations.datasets.from_csv(
    "support_questions.csv",
    name="Support Agent v2",
    acceptance_criteria="Accurate, concise, grounded in docs.",
).publish()

The only required column is query. Optional columns: expected_results, plus expected_capabilities, expected_knowledge_base, and expected_delegations (semicolon-separated lists). Rows with an empty query are skipped with a logged warning. Any extra keyword arguments (criteria, similarity metrics, judge_model, …) are passed through to builder().

query,expected_results
How do I reset my password?,Explain the password reset process step by step.
What payment methods do you accept?,List supported payment methods clearly.

client.evaluations.datasets.from_dataframe(df, name=...) does the same from a pandas DataFrame with the same column contract.

Next steps

Quick Start

Run an evaluation against your new dataset

Evaluation Settings

Reuse one grading config across multiple datasets

EvaluationSubject Fields

Describe the agent under test

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Framework Examples

Programmatic builder

`builder()` parameters

Repetition statistics

`add_case()` parameters

Expected trajectories

Expected retrieval context

Smoke testing

Splits

CSV import

Next steps

Quick Start

Evaluation Settings

EvaluationSubject Fields

​Programmatic builder

​builder() parameters

​Repetition statistics

​add_case() parameters

​Expected trajectories

​Expected retrieval context

​Smoke testing

​Splits

​CSV import

​Next steps

Quick Start

Evaluation Settings

EvaluationSubject Fields

Programmatic builder

`builder()` parameters

Repetition statistics

`add_case()` parameters

Expected trajectories

Expected retrieval context

Smoke testing

Splits

CSV import

Next steps