> ## Documentation Index > Fetch the complete documentation index at: https://developers.agentx.so/llms.txt > Use this file to discover all available pages before exploring further. # Build Dataset > Create evaluation datasets and test cases with the Python SDK A dataset is a named set of test cases plus scoring configuration. Every evaluation run locks a snapshot of the dataset at creation time, so editing a dataset later never changes the scoring of past runs. There are two ways to build one: the fluent builder, or importing from a CSV. ## Programmatic builder ```python theme={null} dataset = ( client.evaluations.datasets .builder( name="Support Agent v2", number_of_requests=3, acceptance_criteria="Accurate, concise, grounded in docs.", rejection_criteria="No hallucinated policies.", ) .add_case( query="How do I reset my password?", expected_results="Explain the password reset process step by step.", ) .add_case( query="What payment methods do you accept?", expected_results="List supported payment methods clearly.", ) .publish() ) print(dataset.id) ``` ### `builder()` parameters | Parameter | Type | Default | Description | | --------------------- | -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `str` | required | Dataset display name | | `description` | `str` | none | Human-readable description | | `number_of_requests` | `int` | `1` | How many times each question is run per evaluation, for consistency testing | | `acceptance_criteria` | `str` | none | What a good response looks like, included in the LLM scoring prompt | | `rejection_criteria` | `str` | none | What a bad response looks like, included in the LLM scoring prompt | | `evaluation_criteria` | `str` | none | Scoring rubric, included in the LLM scoring prompt | | `vector_similarity` | `bool \| dict` | `False` | Enable cosine similarity scoring against `expected_results`, e.g. `{"model": "text-embedding-3-small"}` | | `jaccard_similarity` | `bool` | `False` | Enable Jaccard token-overlap scoring against `expected_results` | | `judge_prompt` | `str` | none | Override the LLM-as-judge's raw prompt template. See [Configuring the judge](/sdk/evaluations/evaluation-settings#configuring-the-judge) | | `judge_model` | `str` | none | Override the LLM-as-judge's model (any OpenAI or Anthropic id from `client.evaluations.list_models()`) | ### `add_case()` parameters | Parameter | Type | Description | | ---------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | `str` | required, the question or prompt sent to the agent | | `expected_results` | `str` | Optional, enables vector and Jaccard similarity scoring for this case | | `judge_guideline` | `str` | Optional extra grading instructions for this specific case, layered on top of the dataset's criteria fields | | `smoke_test_count` | `int` | Optional, 1-10. Ask this question that many extra ways each run (LLM-paraphrased server-side), to catch agents that break on phrasing rather than substance. See [Smoke testing](#smoke-testing) below | | `smoke_test_guidance` | `str` | Optional, only used when `smoke_test_count` is set. Free text steering what kind of variants get generated (tone, adversarial phrasing, different languages, ...) | | `expected_tools` | `list[str]` | Optional. The tool calls a correct run of this case should make - trajectory-matched against the result's linked trace, reported as a pass/fail "Trajectory match" scorer row. See [Expected trajectories](#expected-trajectories) below | | `trajectory_match_mode` | `str` | `"strict"` (default), `"unordered"`, `"superset"`, or `"subset"` - see below | | `expected_retrieval_context` | `str \| list[str]` | Optional. The chunk(s) a correct retriever should fetch for this query - compared against the actually retrieved context with token-level Jaccard similarity, reported as a "Context match (jaccard)" scorer row (0-1), no judge call. See [Expected retrieval context](#expected-retrieval-context) below | `publish()` sends the dataset to AgentX and returns a `Dataset` object with `.id`, which you pass as `dataset_id` to `client.evaluations.run()`. ## Expected trajectories Agentic cases can pin down *how* the answer should be produced, not just what it should say. Declare the expected tool calls; when a result links its trace (return `{"output": ..., "trace_id": span.trace_id}` from your agent function), the engine compares the trace's actual tool-call sequence deterministically: ```python theme={null} .add_case( query="I want a refund for order A-1001, it arrived two weeks ago.", expected_results="Confirms eligibility within the 30-day window.", expected_tools=["lookup_order", "refund_policy"], trajectory_match_mode="unordered", ) ``` | Mode | Passes when | | ----------- | ------------------------------------------------ | | `strict` | Same calls, same order | | `unordered` | Same calls, any order | | `superset` | Every expected call present; extra calls allowed | | `subset` | No unexpected calls; missing some is allowed | The verdict lands on each result as a named scorer row alongside the judge rating - and the LLM judge itself also sees the full trajectory in its prompt, so written criteria about tool behavior are scoreable too. ## Expected retrieval context RAG cases can pin what a correct retriever *should have fetched*. At scoring time the engine compares it against the actually retrieved context - the `retrieval_context` your agent function returned, else the linked trace's retrieval spans - using token-level Jaccard similarity, and reports a **Context match (jaccard)** scorer row (0-1) on each result. It's deterministic and costs no judge call, so it's safe to run on every retriever, chunking, or embedding change: ```python theme={null} .add_case( query="What is the refund window?", expected_retrieval_context=[ "Refund policy: refunds are available within 30 days of delivery for unused items." ], ) ``` The same field is editable in the dashboard (open a dataset case, **Expected retrieval context**, blank line between chunks). Jaccard measures token overlap, not meaning - pair it with the [RAG judge metrics](/evaluation/rag) for semantic verdicts. ## Smoke testing Every variant of a smoke-tested case is graded against the same `expected_results` as the original question. Both the paraphrase text and the count are generated entirely server-side, `.execute()` asks the extra variants and submits them automatically: ```python theme={null} dataset = ( client.evaluations.datasets .builder(name="Support Agent v2") .add_case( query="How do I reset my password?", expected_results="Explain the password reset process step by step.", smoke_test_count=3, smoke_test_guidance="try terse, frustrated, and non-native-speaker phrasing", ) .publish() ) ``` Your agent function can check `case.is_smoke_test_variant` if it wants to handle variants differently, but this isn't required, `case.query` already holds the text to ask either way. Smoke-test results are scored normally but reported as a separate robustness signal, excluded from `report.average_rating` so one hard phrasing doesn't skew your headline score. Only a case's opening question can be smoke-tested, `smoke_test_count`/`smoke_test_guidance` are ignored on follow-up questions. ## CSV import ```python theme={null} dataset = client.evaluations.datasets.from_csv( "support_questions.csv", name="Support Agent v2", acceptance_criteria="Accurate, concise, grounded in docs.", ) ``` The CSV must have `query` and `expected_results` columns. An optional `case_id` column lets you re-run the same file idempotently: matching rows update existing cases instead of creating duplicates. ```csv theme={null} case_id,query,expected_results pw-reset,How do I reset my password?,Explain the password reset process step by step. payment-methods,What payment methods do you accept?,List supported payment methods clearly. ``` ## Next steps Run an evaluation against your new dataset Reuse one grading config across multiple datasets Describe the agent under test