Build Dataset

A dataset is a named set of test cases plus scoring configuration. Every evaluation run locks a snapshot of the dataset at creation time, so editing a dataset later never changes the scoring of past runs. There are two ways to build one: the fluent builder, or importing from a CSV.

Programmatic builder

dataset = (
    client.evaluations.datasets
    .builder(
        name="Support Agent v2",
        number_of_requests=3,
        acceptance_criteria="Accurate, concise, grounded in docs.",
        rejection_criteria="No hallucinated policies.",
    )
    .add_case(
        query="How do I reset my password?",
        expected_results="Explain the password reset process step by step.",
    )
    .add_case(
        query="What payment methods do you accept?",
        expected_results="List supported payment methods clearly.",
    )
    .publish()
)

print(dataset.id)

`builder()` parameters

Parameter	Type	Default	Description
`name`	`str`	required	Dataset display name
`description`	`str`	none	Human-readable description
`number_of_requests`	`int`	`1`	How many times each question is run per evaluation, for consistency testing
`acceptance_criteria`	`str`	none	What a good response looks like, included in the LLM scoring prompt
`rejection_criteria`	`str`	none	What a bad response looks like, included in the LLM scoring prompt
`evaluation_criteria`	`str`	none	Scoring rubric, included in the LLM scoring prompt
`vector_similarity`	`bool \| dict`	`False`	Enable cosine similarity scoring against `expected_results`, e.g. `{"model": "text-embedding-3-small"}`
`jaccard_similarity`	`bool`	`False`	Enable Jaccard token-overlap scoring against `expected_results`
`judge_prompt`	`str`	none	Override the LLM-as-judge’s raw prompt template. See Configuring the judge
`judge_model`	`str`	none	Override the LLM-as-judge’s model (any OpenAI or Anthropic id from `client.evaluations.list_models()`)

`add_case()` parameters

Parameter	Type	Description
`query`	`str`	required, the question or prompt sent to the agent
`expected_results`	`str`	Optional, enables vector and Jaccard similarity scoring for this case
`judge_guideline`	`str`	Optional extra grading instructions for this specific case, layered on top of the dataset’s criteria fields
`smoke_test_count`	`int`	Optional, 1-10. Ask this question that many extra ways each run (LLM-paraphrased server-side), to catch agents that break on phrasing rather than substance. See Smoke testing below
`smoke_test_guidance`	`str`	Optional, only used when `smoke_test_count` is set. Free text steering what kind of variants get generated (tone, adversarial phrasing, different languages, …)
`expected_tools`	`list[str]`	Optional. The tool calls a correct run of this case should make - trajectory-matched against the result’s linked trace, reported as a pass/fail “Trajectory match” scorer row. See Expected trajectories below
`trajectory_match_mode`	`str`	`"strict"` (default), `"unordered"`, `"superset"`, or `"subset"` - see below
`expected_retrieval_context`	`str \| list[str]`	Optional. The chunk(s) a correct retriever should fetch for this query - compared against the actually retrieved context with token-level Jaccard similarity, reported as a “Context match (jaccard)” scorer row (0-1), no judge call. See Expected retrieval context below

publish() sends the dataset to AgentX and returns a Dataset object with .id, which you pass as dataset_id to client.evaluations.run().

Expected trajectories

Agentic cases can pin down how the answer should be produced, not just what it should say. Declare the expected tool calls; when a result links its trace (return {"output": ..., "trace_id": span.trace_id} from your agent function), the engine compares the trace’s actual tool-call sequence deterministically:

.add_case(
    query="I want a refund for order A-1001, it arrived two weeks ago.",
    expected_results="Confirms eligibility within the 30-day window.",
    expected_tools=["lookup_order", "refund_policy"],
    trajectory_match_mode="unordered",
)

Mode	Passes when
`strict`	Same calls, same order
`unordered`	Same calls, any order
`superset`	Every expected call present; extra calls allowed
`subset`	No unexpected calls; missing some is allowed

The verdict lands on each result as a named scorer row alongside the judge rating - and the LLM judge itself also sees the full trajectory in its prompt, so written criteria about tool behavior are scoreable too.

Expected retrieval context

RAG cases can pin what a correct retriever should have fetched. At scoring time the engine compares it against the actually retrieved context - the retrieval_context your agent function returned, else the linked trace’s retrieval spans - using token-level Jaccard similarity, and reports a Context match (jaccard) scorer row (0-1) on each result. It’s deterministic and costs no judge call, so it’s safe to run on every retriever, chunking, or embedding change:

.add_case(
    query="What is the refund window?",
    expected_retrieval_context=[
        "Refund policy: refunds are available within 30 days of delivery for unused items."
    ],
)

The same field is editable in the dashboard (open a dataset case, Expected retrieval context, blank line between chunks). Jaccard measures token overlap, not meaning - pair it with the RAG judge metrics for semantic verdicts.

Smoke testing

Every variant of a smoke-tested case is graded against the same expected_results as the original question. Both the paraphrase text and the count are generated entirely server-side, .execute() asks the extra variants and submits them automatically:

dataset = (
    client.evaluations.datasets
    .builder(name="Support Agent v2")
    .add_case(
        query="How do I reset my password?",
        expected_results="Explain the password reset process step by step.",
        smoke_test_count=3,
        smoke_test_guidance="try terse, frustrated, and non-native-speaker phrasing",
    )
    .publish()
)

Your agent function can check case.is_smoke_test_variant if it wants to handle variants differently, but this isn’t required, case.query already holds the text to ask either way. Smoke-test results are scored normally but reported as a separate robustness signal, excluded from report.average_rating so one hard phrasing doesn’t skew your headline score. Only a case’s opening question can be smoke-tested, smoke_test_count/smoke_test_guidance are ignored on follow-up questions.

CSV import

dataset = client.evaluations.datasets.from_csv(
    "support_questions.csv",
    name="Support Agent v2",
    acceptance_criteria="Accurate, concise, grounded in docs.",
)

The CSV must have query and expected_results columns. An optional case_id column lets you re-run the same file idempotently: matching rows update existing cases instead of creating duplicates.

case_id,query,expected_results
pw-reset,How do I reset my password?,Explain the password reset process step by step.
payment-methods,What payment methods do you accept?,List supported payment methods clearly.

Next steps

Quick Start

Run an evaluation against your new dataset

Evaluation Settings

Reuse one grading config across multiple datasets

EvaluationSubject Fields

Describe the agent under test

Pre-Deployment Quality Gate

Build & Run

Scoring & Datasets

Framework Examples

Programmatic builder

`builder()` parameters

`add_case()` parameters

Expected trajectories

Expected retrieval context

Smoke testing

CSV import

Next steps

Quick Start

Evaluation Settings

EvaluationSubject Fields

​Programmatic builder

​builder() parameters

​add_case() parameters

​Expected trajectories

​Expected retrieval context

​Smoke testing

​CSV import

​Next steps

Quick Start

Evaluation Settings

EvaluationSubject Fields

Programmatic builder

`builder()` parameters

`add_case()` parameters

Expected trajectories

Expected retrieval context

Smoke testing

CSV import

Next steps