CI/CD Overview

CI/CD evaluation on AgentX is an ordinary custom evaluation run plus one extra call: after the run finalizes, GET /runs/:id/gate turns its ratings into a binary passed / failed verdict your pipeline can exit on. Your agent runs in your infrastructure; the engine scores results and computes the gate.

There is no separate CI run type or /ci-runs endpoint family on the self-host engine - a CI run is a normal evaluation run, so it appears in Evaluate → Runs with full results, analysis, and history like any other. (The hosted platform additionally offers a lightweight CI Ingest API, which the SDK’s run_eval() wraps - see SDK CI/CD.) The Python SDK wraps the whole self-host flow too, including report.gate(...) and the pytest helper agentx.testing.assert_evaluation.

How it works

CI pipeline
    │
    ├─ POST /custom-agent-evaluations/runs               ← create the run
    │
    ├─ (run your agent per dataset case)
    │
    ├─ POST /custom-agent-evaluations/runs/:id/results   ← submit + judge-score each batch
    │
    ├─ POST /custom-agent-evaluations/runs/:id/finalize  ← close the run
    │
    └─ GET  /custom-agent-evaluations/runs/:id/gate      ← verdict: passed true/false

Gate checks

The gate runs one or both of these checks (at least one is required) and passes only when every requested check passes:

Absolute floor (failUnder=<0-10>): the run’s average rating must be at or above the floor. A run with no rated results fails this check.
No regression (noRegression=true): the average must not have dropped more than tolerance (default 0.5) below the baseline run’s average. The baseline is the dataset’s most recent earlier completed rated run with the same grading identity - same grading config (evaluationSettingsId), same scorer group, and same split - so switching how a run is graded starts a fresh baseline rather than comparing incomparable scores. Judge scores are noisy, so an exact comparison would flake builds on variance; tune tolerance to your dataset. With no matching prior run, the check passes explicitly (“nothing to regress against”). Single-trace evaluations are never used as the baseline.

The verdict is computed fresh from stored ratings on every call, so re-running a failed CI job re-evaluates against current state.

Gate history and alerting

record=true appends the verdict to the project’s gate history - what the dashboard’s CI page lists - along with an optional caller label (e.g. "github-actions"). A recorded failed gate also fires the webhook channels configured on the project’s monitoring profiles, so a gate blocking a merge can reach Slack. Preview calls omit record and stay compute-only.

Smoke splits

Tag a subset of dataset cases with a named split (main_question.splits: ["smoke"]) and create the run with "split": "smoke" to gate PRs on a fast subset while nightly jobs run the full dataset. Case indexes are preserved, so per-case comparisons line up across split and full runs.

Endpoints

Method	Path	Description
`POST`	`/custom-agent-evaluations/runs`	Create the run
`GET`	`/custom-agent-evaluations/datasets/:id`	Read the dataset’s test cases
`POST`	`/custom-agent-evaluations/runs/:id/results`	Submit results
`POST`	`/custom-agent-evaluations/runs/:id/finalize`	Finalize the run
`GET`	`/custom-agent-evaluations/runs/:id/gate`	Compute the gate verdict

All endpoints share the engine’s global error envelope for authentication (401) and rate limiting (429) - see Errors.

Minimal pipeline example

BASE=http://localhost:4700/api/v1/custom-agent-evaluations
KEY="x-api-key: $AGENTX_API_KEY"

RUN_ID=$(curl -s -X POST "$BASE/runs" -H "$KEY" -H "Content-Type: application/json" \
  -d '{"datasetId":"'"$DATASET_ID"'","runSource":"sdk"}' | jq -r .runId)

# ... run your agent per case and POST batches to $BASE/runs/$RUN_ID/results ...

curl -s -X POST "$BASE/runs/$RUN_ID/finalize" -H "$KEY" > /dev/null

PASSED=$(curl -s "$BASE/runs/$RUN_ID/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
  -H "$KEY" | jq -r .passed)

[ "$PASSED" = "true" ] || { echo "CI gate FAILED"; exit 1; }

Tracing

Custom Evaluations

CI Gate

Agent Registry

CI/CD Overview

How it works

Gate checks

Gate history and alerting

Smoke splits

Endpoints

Minimal pipeline example

​How it works

​Gate checks

​Gate history and alerting

​Smoke splits

​Endpoints

​Minimal pipeline example

How it works

Gate checks

Gate history and alerting

Smoke splits

Endpoints

Minimal pipeline example