> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Dataset

> Create an evaluation dataset with questions and scoring configuration

Creates a dataset: the questions your agent will be evaluated against plus the grading
configuration (judge criteria, similarity metrics, code scorers) used when a run doesn't bring
its own [evaluation-settings config](/api-reference/custom-eval/overview#endpoints). Creation
also seeds version `v0` of the dataset's version history.

## Authentication

<ParamField header="x-api-key" type="string" required>
  Project API key.
</ParamField>

## Body

<ParamField body="name" type="string" required>
  Dataset display name.
</ParamField>

<ParamField body="questions" type="array" default="[]">
  Question objects. Not validated at creation time (an empty or malformed array is accepted),
  but a run against a dataset with no usable questions has nothing to score. Each question
  wraps its fields in `main_question`:

  ```json theme={null}
  [
    {
      "main_question": {
        "query": "How do I reset my password?",
        "expectedResults": "The user should click Forgot Password on the login screen."
      }
    }
  ]
  ```

  <Expandable title="main_question fields">
    | Field                      | Type                | Description                                                                                                                                                       |
    | -------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `query`                    | string              | The question posed to the agent                                                                                                                                   |
    | `expectedResults`          | string              | Reference answer. Enables similarity scoring and is shown to the judge; required per-case by `requiresExpected` grading configs                                   |
    | `judgeGuideline`           | string              | Case-specific guidance added to the judge prompt                                                                                                                  |
    | `retrievalContext`         | string              | Statically pinned context for `{context}`-referencing judge prompts (used only when the result and its linked trace carry none)                                   |
    | `expectedRetrievalContext` | string \| string\[] | Reference retrieval: what a correct retriever should have fetched. Compared to the actually-retrieved context by token Jaccard similarity - no judge call         |
    | `expectedTrajectory`       | object              | `{ "tools": string[], "mode": "strict" \| "unordered" \| "subset" \| "superset" }` - expected tool calls, matched against the linked trace's actual tool sequence |
    | `smokeTest`                | object              | `{ "enabled": boolean, "count": number, "guidance": string }` - generate paraphrase variants of this question at run creation for consistency testing             |
    | `splits`                   | string\[]           | Named subset tags; a run created with `split` covers only cases tagged with it                                                                                    |
  </Expandable>
</ParamField>

<ParamField body="description" type="string">
  Human-readable description.
</ParamField>

<ParamField body="numberOfRequests" type="number" default="1">
  How many times each question is run per evaluation (for consistency testing).
</ParamField>

<ParamField body="acceptanceCriteria" type="string">
  What a good response looks like. Included in the LLM scoring prompt.
</ParamField>

<ParamField body="rejectionCriteria" type="string">
  What a bad response looks like. Included in the LLM scoring prompt.
</ParamField>

<ParamField body="evaluationCriteria" type="string">
  Scoring rubric. Included in the LLM scoring prompt.
</ParamField>

<ParamField body="vectorSimilarity" type="object">
  Enable cosine similarity scoring against `expectedResults` (requires an embeddings-capable
  provider key). Only `{ "enabled": true }` activates it; `model` optionally selects the
  embedding model.

  ```json theme={null}
  { "enabled": true, "model": "text-embedding-3-small" }
  ```
</ParamField>

<ParamField body="jaccardSimilarity" type="object">
  Enable Jaccard token-overlap scoring against `expectedResults`: `{ "enabled": true }`.
</ParamField>

<ParamField body="bleuScore" type="object">
  Enable BLEU scoring against `expectedResults`: `{ "enabled": true }`.
</ParamField>

<ParamField body="rougeScore" type="object">
  Enable ROUGE scoring against `expectedResults`: `{ "enabled": true }`.
</ParamField>

<ParamField body="codeScorers" type="array">
  Deterministic code scorers run on every result. Each:
  `{ "id"?: string, "name": string, "code": string, "enabled"?: boolean }`. Entries with empty
  `code` are dropped; `enabled` defaults to `true`.
</ParamField>

## Response

Returns `201 Created` with the full dataset document, keyed by `_id`. `status` is always
`"published"` on self-host. The enabled similarity metrics come back as top-level keys
(`vectorSimilarity`, `jaccardSimilarity`, ...), matching what was sent.

## Errors

| Status | Body                              | Meaning                                |
| ------ | --------------------------------- | -------------------------------------- |
| `400`  | `{ "error": "name is required" }` | `name` missing, not a string, or blank |

<RequestExample>
  ```bash cURL theme={null}
  curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \
    -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "Customer Support Q3 2026",
      "description": "Core support questions",
      "numberOfRequests": 3,
      "acceptanceCriteria": "Accurate, empathetic, resolves the issue",
      "rejectionCriteria": "Hallucinates, ignores the question",
      "questions": [
        {
          "main_question": {
            "query": "How do I reset my password?",
            "expectedResults": "Click Forgot Password on the login screen."
          }
        },
        {
          "main_question": {
            "query": "What payment methods do you accept?"
          }
        }
      ],
      "vectorSimilarity": { "enabled": true },
      "jaccardSimilarity": { "enabled": true }
    }'
  ```

  ```python Python SDK theme={null}
  dataset = (
      client.evaluations.datasets.builder(
          "Customer Support Q3 2026",
          description="Core support questions",
          number_of_requests=3,
          acceptance_criteria="Accurate, empathetic, resolves the issue",
          rejection_criteria="Hallucinates, ignores the question",
          vector_similarity=True,
          jaccard_similarity=True,
      )
      .add_case("How do I reset my password?",
                expected_results="Click Forgot Password on the login screen.")
      .add_case("What payment methods do you accept?")
      .publish()
  )
  ```
</RequestExample>

<ResponseExample>
  ```json 201 Created theme={null}
  {
    "_id": "dS4tG7hNb2VxZ8kQ5wMyA",
    "name": "Customer Support Q3 2026",
    "description": "Core support questions",
    "numberOfRequests": 3,
    "vectorSimilarity": { "enabled": true },
    "jaccardSimilarity": { "enabled": true },
    "acceptanceCriteria": "Accurate, empathetic, resolves the issue",
    "rejectionCriteria": "Hallucinates, ignores the question",
    "questions": [
      { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } },
      { "main_question": { "query": "What payment methods do you accept?" } }
    ],
    "status": "published",
    "createdAt": "2026-08-27T10:00:00.000Z"
  }
  ```

  ```json 400 Missing name theme={null}
  {
    "error": "name is required"
  }
  ```
</ResponseExample>