> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Create CI Run

> Start the evaluation run a CI pipeline will gate on

A CI run **is** a standard evaluation run - this is the same endpoint as
[Create Run](/api-reference/custom-eval/create-run), shown here in its CI framing. Create the
run at the start of the pipeline job, drive your agent through the dataset's cases, submit and
finalize, then compute the [gate](/api-reference/ci-cd/get-gate).

Two fields matter most in CI:

* **`evaluationSubject.version`** (or `evaluationSubject.metadata.version`): tag the run with
  the branch or commit under test, so version-to-version comparisons in the dashboard line up
  with your git history.
* **`split`**: run only the dataset cases tagged with a named split (e.g. `"smoke"`) for fast
  PR gates, keeping the full dataset for nightly runs.

## Authentication

<ParamField header="x-api-key" type="string" required>
  Project API key (use a CI secret, never source control).
</ParamField>

## Body

<ParamField body="datasetId" type="string" required>
  Dataset ID to evaluate against.
</ParamField>

<ParamField body="evaluationSubject" type="object">
  Metadata about the agent under test. In CI, include a `version` tag:

  ```json theme={null}
  {
    "kind": "custom_agent",
    "displayName": "Customer Support Bot",
    "version": "feat/new-retrieval@a1b2c3d4"
  }
  ```
</ParamField>

<ParamField body="evaluationSettingsId" type="string">
  Grade with a standalone grading config instead of the dataset's own criteria.
</ParamField>

<ParamField body="runSource" type="string" default="&#x22;sdk&#x22;">
  How the run was triggered.
</ParamField>

<ParamField body="split" type="string">
  Named case subset to cover (cases tagged via `main_question.splits`). The caller filters
  which cases it executes; the split is recorded on the run's subject.
</ParamField>

## Response

Returns `201 Created` with `{ runId, datasetId, status: "in_progress", smokeTestVariants }` -
see [Create Run](/api-reference/custom-eval/create-run) for the full field reference. Read the
cases to execute from the dataset itself
([Get Test Cases](/api-reference/ci-cd/get-test-cases)).

## Errors

| Status | Body                                                   | Meaning                                                   |
| ------ | ------------------------------------------------------ | --------------------------------------------------------- |
| `400`  | `{ "error": "datasetId is required" }`                 | Missing or non-string `datasetId`                         |
| `400`  | `{ "error": "evaluationSettingsId must be a string" }` | `evaluationSettingsId` present but not a string (or null) |
| `400`  | `{ "error": "scorerGroupId must be a string" }`        | `scorerGroupId` present but not a string (or null)        |
| `404`  | `{ "error": "Scorer group not found" }`                | No scorer group with this id in the project               |
| `404`  | `{ "error": "Dataset not found" }`                     | No dataset with this id in the project                    |

<RequestExample>
  ```bash cURL theme={null}
  curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
    -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
    -H "Content-Type: application/json" \
    -d '{
      "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
      "evaluationSubject": {
        "kind": "custom_agent",
        "displayName": "Customer Support Bot",
        "version": "feat/new-retrieval@a1b2c3d4"
      },
      "split": "smoke"
    }'
  ```

  ```python Python SDK theme={null}
  import sys

  report = (
      client.evaluations.run(
          "dS4tG7hNb2VxZ8kQ5wMyA",
          subject={"kind": "custom_agent", "displayName": "Customer Support Bot",
                   "metadata": {"version": "feat/new-retrieval@a1b2c3d4"}},
          split="smoke",
      )
      .execute(my_agent)   # called once per case
      .finalize()
  )
  gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
  if not gate.passed:
      sys.exit(1)
  ```
</RequestExample>

<ResponseExample>
  ```json 201 Created theme={null}
  {
    "runId": "rK7dP2qWx9TzB4mV6nJcE",
    "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
    "status": "in_progress",
    "smokeTestVariants": null
  }
  ```
</ResponseExample>