> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Run

> Start a new evaluation run against a dataset

Initialises a new evaluation run with status `"in_progress"`. Submit your agent's outputs to it
with [Submit Results](/api-reference/custom-eval/submit-results), then
[finalize](/api-reference/custom-eval/finalize-run). If any dataset case has smoke testing
enabled, the paraphrase variants are generated once here and frozen for the run's lifetime.

## Authentication

<ParamField header="x-api-key" type="string" required>
  Project API key.
</ParamField>

## Body

<ParamField body="datasetId" type="string" required>
  Dataset ID to evaluate against.
</ParamField>

<ParamField body="evaluationSettingsId" type="string">
  Grade with a standalone, reusable grading config (criteria, judge prompt/model, similarity
  metrics, code scorers) instead of the dataset's own configuration. Lets one config run
  against any dataset. Omit to use the dataset's own criteria.
</ParamField>

<ParamField body="scorerGroupId" type="string">
  Grade with a [scorer group](/monitor/scorer-groups) - the group's weighted aggregate fills
  the run's rating, judge members land as per-row verdicts, and deterministic members as scorer
  rows. A group is a complete grading recipe: when it resolves, `evaluationSettingsId` and
  `additionalScorerIds` are ignored.
</ParamField>

<ParamField body="additionalScorerIds" type="string[]">
  Extra LLM judge scorer ids (up to 4, self-host) that also grade every result - one agent
  execution, N verdicts. The primary scorer keeps the `rating` column; each extra verdict lands
  in the result row's `judgeScorerResults` and rolls up into the run's `scorerBreakdown`
  ([Get Run](/api-reference/custom-eval/get-run)). Duplicates of the primary are dropped, and
  an id that no longer resolves to a scorer is silently skipped (that scorer simply does not
  grade the run) - unlike `scorerGroupId`, where an unknown id is a hard `404`.
</ParamField>

<ParamField body="evaluationSubject" type="object">
  Free-form metadata about the agent being evaluated (e.g. `kind`, `displayName`, `framework`,
  `runtime`, `agentInstructions`). Stored on the run and echoed back by
  [Get Run](/api-reference/custom-eval/get-run). A `version` field (or
  `metadata.version`) tags the run for version-to-version comparison in the dashboard.
</ParamField>

<ParamField body="runSource" type="string" default="&#x22;sdk&#x22;">
  How the run was triggered, e.g. `"sdk"`. Trace evaluations use `"trace-eval"` internally and
  are excluded from CI-gate baselines.
</ParamField>

<ParamField body="sdk" type="object">
  SDK name/version info, stored for diagnostics.
</ParamField>

<ParamField body="split" type="string">
  Named case subset: run only the dataset cases tagged with this split
  (`main_question.splits`). The caller does the filtering when executing; the split name is
  recorded on the run's subject so split runs are visibly split runs. Original question
  indexes are preserved, so per-case comparisons line up with full runs.
</ParamField>

## Response

Returns `201 Created`.

<ResponseField name="runId" type="string">
  Use in all subsequent calls for this run.
</ResponseField>

<ResponseField name="datasetId" type="string">
  Echoed back.
</ResponseField>

<ResponseField name="status" type="string">
  Always `"in_progress"` on creation.
</ResponseField>

<ResponseField name="smokeTestVariants" type="array | null">
  One group per dataset case with `smokeTest.enabled`:
  `[{ "questionIndex": number, "variants": string[] }]`. `null` when no case requests smoke
  testing. Run each variant like a normal case and submit its result with
  `isSmokeTestVariant: true`.
</ResponseField>

## Errors

| Status | Body                                                   | Meaning                                                                                                                                                               |
| ------ | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400`  | `{ "error": "datasetId is required" }`                 | Missing or non-string `datasetId`                                                                                                                                     |
| `400`  | `{ "error": "evaluationSettingsId must be a string" }` | `evaluationSettingsId` present but not a string (or null)                                                                                                             |
| `400`  | `{ "error": "scorerGroupId must be a string" }`        | `scorerGroupId` present but not a string (or null) - a 400, not a silent drop, so a malformed group id can't downgrade the run to the dataset's own grading unnoticed |
| `404`  | `{ "error": "Scorer group not found" }`                | No scorer group with this id in the project                                                                                                                           |
| `404`  | `{ "error": "Dataset not found" }`                     | No dataset with this id in the project                                                                                                                                |

<RequestExample>
  ```bash cURL theme={null}
  curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
    -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
    -H "Content-Type: application/json" \
    -d '{
      "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
      "evaluationSubject": {
        "kind": "custom_agent",
        "displayName": "Customer Support Bot",
        "framework": "langchain",
        "version": "v2-retrieval"
      },
      "runSource": "sdk"
    }'
  ```

  ```python Python SDK theme={null}
  from agentx.evaluations.models import EvaluationSubject

  # client.evaluations.run(dataset_id, subject, ...) drives the whole lifecycle;
  # init_run alone creates the run to drive manually.
  run = client.evaluations.init_run(
      dataset_id="dS4tG7hNb2VxZ8kQ5wMyA",
      subject=EvaluationSubject(kind="custom_agent", display_name="Customer Support Bot"),
  )
  ```
</RequestExample>

<ResponseExample>
  ```json 201 Created theme={null}
  {
    "runId": "rK7dP2qWx9TzB4mV6nJcE",
    "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
    "status": "in_progress",
    "smokeTestVariants": null
  }
  ```

  ```json 404 Dataset not found theme={null}
  {
    "error": "Dataset not found"
  }
  ```
</ResponseExample>