CI/CD Overview

CI/CD evaluation on AgentX is an ordinary custom evaluation run plus one extra call: after the run finalizes, GET /runs/:id/gate turns its ratings into a binary passed / failed verdict your pipeline can exit on. Your agent runs in your infrastructure; the engine scores results and computes the gate.

There is no separate CI run type or /ci-runs endpoint family on the self-host engine - a CI run is a normal evaluation run, so it appears in Evaluate > Runs with full results, analysis, and history like any other. (The hosted platform additionally offers a lightweight CI Ingest API, which the SDK’s run_eval() wraps - see SDK CI/CD.) The Python SDK wraps the whole self-host flow too, including report.gate(...) and the pytest helper agentx.testing.assert_evaluation.

How it works

CI pipeline
    │
    ├─ POST /custom-agent-evaluations/runs               ← create the run
    │
    ├─ (run your agent per dataset case)
    │
    ├─ POST /custom-agent-evaluations/runs/:id/results   ← submit + judge-score each batch
    │
    ├─ POST /custom-agent-evaluations/runs/:id/finalize  ← close the run
    │
    └─ GET  /custom-agent-evaluations/runs/:id/gate      ← verdict: passed true/false

Gate checks

The gate runs one or both of these checks (at least one is required) and passes only when every requested check passes:

Absolute floor (failUnder=<0-10>): the run’s average rating must be at or above the floor. A run with no rated results fails this check.
No regression (noRegression=true): the average must not have dropped more than tolerance (default 0.5) below the dataset’s previous completed run. Judge scores are noisy, so an exact comparison would flake builds on variance; tune tolerance to your dataset. With no prior completed rated run, the check passes explicitly (“nothing to regress against”). Single-trace evaluations are never used as the baseline.

The verdict is computed fresh from stored ratings on every call, so re-running a failed CI job re-evaluates against current state.

Gate history and alerting

record=true appends the verdict to the project’s gate history - what the dashboard’s CI page lists - along with an optional caller label (e.g. "github-actions"). A recorded failed gate also fires the webhook channels configured on the project’s monitoring profiles, so a gate blocking a merge can reach Slack. Preview calls omit record and stay compute-only.

Smoke splits

Tag a subset of dataset cases with a named split (main_question.splits: ["smoke"]) and create the run with "split": "smoke" to gate PRs on a fast subset while nightly jobs run the full dataset. Case indexes are preserved, so per-case comparisons line up across split and full runs.

Endpoints

Method	Path	Description
`POST`	`/custom-agent-evaluations/runs`	Create the run
`GET`	`/custom-agent-evaluations/datasets/:id`	Read the dataset’s test cases
`POST`	`/custom-agent-evaluations/runs/:id/results`	Submit results
`POST`	`/custom-agent-evaluations/runs/:id/finalize`	Finalize the run
`GET`	`/custom-agent-evaluations/runs/:id/gate`	Compute the gate verdict

Minimal pipeline example

BASE=http://localhost:4700/api/v1/custom-agent-evaluations
KEY="x-api-key: $AGENTX_API_KEY"

RUN_ID=$(curl -s -X POST "$BASE/runs" -H "$KEY" -H "Content-Type: application/json" \
  -d '{"datasetId":"'"$DATASET_ID"'","runSource":"sdk"}' | jq -r .runId)

# ... run your agent per case and POST batches to $BASE/runs/$RUN_ID/results ...

curl -s -X POST "$BASE/runs/$RUN_ID/finalize" -H "$KEY" > /dev/null

PASSED=$(curl -s "$BASE/runs/$RUN_ID/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
  -H "$KEY" | jq -r .passed)

[ "$PASSED" = "true" ] || { echo "CI gate FAILED"; exit 1; }

Tracing

Custom Evaluations

CI Gate

Agent Registry

CI/CD Overview

How it works

Gate checks

Gate history and alerting

Smoke splits

Endpoints

Minimal pipeline example

​How it works

​Gate checks

​Gate history and alerting

​Smoke splits

​Endpoints

​Minimal pipeline example

How it works

Gate checks

Gate history and alerting

Smoke splits

Endpoints

Minimal pipeline example