CI/CD Evaluation

The CI/CD evaluation feature lets you run your agent against a test dataset in your CI pipeline and receive a binary PASS / FAIL gate result. If the gate fails, the pipeline exits with a non-zero code and blocks the merge or deploy.

Two backends, two flows. run_eval() and the create_ci_run family below use the hosted platform’s CI Ingest API (/ingest/ci-runs) and 404 against a self-host engine. On self-host, a CI run is an ordinary evaluation run plus the run gate - report.gate(...) / GET /runs/:id/gate - documented in the API reference.

Prerequisites

Create an evaluation dataset in AgentX with at least one question.
Enable CI/CD in the dataset settings and set a pass rate threshold.
Export AGENTX_API_KEY in your environment.

High-level: `run_eval()`

The easiest path: one call handles the entire lifecycle:

from agentx import AgentX, CIGateFailure

client = AgentX.from_env()

def my_agent(query: str) -> str:
    # your agent here
    return call_my_agent(query)

result = client.tracer.run_eval(
    dataset_id="6876ddd222bbb333ccc444ee",
    agent_fn=my_agent,
    agent_name="customer-support-agent",
    # Keys must be camelCase. The API's gitContext schema is strict
    # and silently drops unrecognized (e.g. snake_case) keys.
    git_context={
        "branch": "feat/new-retrieval",
        "commitSha": "a1b2c3d4e5f6",
    },
    fail_on_gate=True,   # raises CIGateFailure if gate is "fail"
)

print(f"Gate: {result.gate}  |  Pass rate: {result.pass_rate:.0%}")

Parameters

Parameter	Type	Default	Description
`dataset_id`	`str`	required	Dataset id; its grading config must have `ci.enabled: true`
`agent_fn`	`Callable[[str], str]`	required	Function that takes a query and returns a response
`agent_name`	`str`	none	Label for this agent on the AgentX platform
`pass_rate_threshold`	`float`	dataset default	Per-run override (0.0-1.0)
`git_context`	`dict`	none	Branch, commit SHA, PR number, etc.
`concurrency`	`int`	`1`	Max parallel question invocations
`fail_on_gate`	`bool`	`False`	Raise `CIGateFailure` if gate is `"fail"`
`timeout_per_question`	`float`	none	Seconds before a question times out

Return: `CIRunResult`

@dataclass
class CIRunResult:
    run_id: str
    gate: Literal["pass", "fail"]
    pass_rate: float                     # e.g. 0.875
    total_questions: int
    passed_questions: int
    scores: list[CIQuestionScore]
    violations: list[ThresholdViolation]
    git_context: dict | None            # echoed back from create (branch, commit SHA, ...)
    finalized_at: str | None

Low-level: step-by-step

For custom orchestration: parallel execution, streaming, or external agents:

from agentx import AgentX
from agentx.tracing import CIRun, CIRunResult

client = AgentX.from_env()
tracer = client.tracer

# 1. Create the run, receive test cases
run: CIRun = tracer.create_ci_run(
    dataset_id="6876ddd222bbb333ccc444ee",
    agent_name="my-agent",
    git_context={"branch": "main", "commitSha": "abc1234"},
)

# 2. Run agent against each test case
for tc in run.test_cases:
    output = my_agent(tc.query or "")       # tc.query is None if exposeTestInputs is off
    score = tracer.submit_result(
        run.run_id,
        tc.index,
        output,
        latency_ms=350,
    )
    if score.gate_fired:
        print("failFast triggered, run already finalized as FAIL")
        break

# 3. Finalize and get the gate decision
result: CIRunResult = tracer.finalize_ci_run(run.run_id)
print(f"Gate: {result.gate}  ({result.passed_questions}/{result.total_questions} passed)")

Exception handling

from agentx import AgentX, CIGateFailure, CINotEnabled, DatasetNotFound

try:
    result = client.tracer.run_eval(
        dataset_id=dataset_id,
        agent_fn=my_agent,
        fail_on_gate=True,
    )
except DatasetNotFound:
    print("Dataset not found, check the dataset_id")
    sys.exit(1)
except CINotEnabled:
    print("CI not enabled on this dataset, enable it in dataset settings")
    sys.exit(1)
except CIGateFailure as e:
    result = e.result
    print(f"Gate FAILED, {result.pass_rate:.0%} passed")
    for v in result.violations:
        print(f"  Q{v.question_index}: {v.metric} was {v.actual:.2f}, threshold {v.threshold:.2f}")
    sys.exit(1)

Parallel question execution

Run multiple questions concurrently to speed up large datasets:

result = client.tracer.run_eval(
    dataset_id=dataset_id,
    agent_fn=my_agent,
    concurrency=4,       # run 4 questions in parallel
    fail_on_gate=True,
)

Inspecting scores

for score in result.scores:
    status = "✓" if score.passed else "✗"
    print(f"  [{status}] Q{score.question_index}: {score.rating}/10, {score.justification[:80]}")

Polling a run

If you submit results asynchronously, poll until finalized:

import time
from agentx.tracing import CIRunStatus

while True:
    status: CIRunStatus = tracer.get_ci_run(run_id)
    if status.status in ("completed", "failed"):
        break
    time.sleep(5)

print(f"Gate: {status.gate}")

GitHub Actions

See the GitHub Actions integration guide for a complete workflow template.

Gating a Custom Agent Evaluation run

The flow above is the dedicated CI/CD run type (pass-rate gate over per-question pass/fail). A regular Custom Agent Evaluation run can be gated too, on its average judge rating - useful when the same golden dataset serves both deep evaluation and the merge gate (self-host):

Set AGENTX_EVAL_QUIET=1 in CI to silence this runner’s interactive progress UI (spinners, per-case lines) - gate verdicts, results, and errors still print, keeping a gated run’s log to a few lines. It applies to the self-host custom-agent runner only, not tracer.run_eval.

import sys

run = (
    client.evaluations
    .run(dataset_id=DATASET_ID, subject=SUBJECT)
    .execute(my_agent)
    .finalize()
)

gate = run.gate(fail_under=7, no_regression=True)
sys.exit(gate.exit_code)   # 0 on pass, 1 on fail

Parameter	Type	Default	Description
`fail_under`	`float`	none	Fail when the run’s average rating is below this floor (0-10)
`no_regression`	`bool`	`False`	Fail when the average dropped more than `tolerance` below the dataset’s previous completed run
`tolerance`	`float`	`0.5`	Allowed drop for `no_regression` before it counts as a regression (judge scores are noisy)
`caller`	`str`	`"sdk"`	Free label recorded in the gate history, e.g. your CI job name
`scorer`	`str`	none	Gate a named additional judge scorer’s average instead of the primary rating (id or name, e.g. `scorer="Safety"`) - an unknown name is a 400, and only judge scorers resolve (pattern/code group members have no per-run average)

For no_regression, the baseline must have measured the same thing: only completed runs with the same evaluationSettingsId, scorerGroupId, and split qualify, trace evaluations are excluded, and only the 5 newest candidates are considered. With no matching baseline the check passes explicitly (“nothing to regress against”). At least one check is required. gate() prints a CI-log-friendly per-check verdict and returns a GateResult with .passed, .exit_code (0/1), .average_rating, .baseline_average, .baseline_run_id, and the per-check .checks list; the caller decides whether to exit. Every recorded verdict lands in the dashboard’s CI Gates history. Two standalone forms cover runs created elsewhere:

gate = client.evaluations.gate_run(run_id, fail_under=7, record=False)  # preview, not recorded
client.evaluations.list_gates()   # recorded verdicts, newest first

pytest-native assertions

agentx.testing.assert_evaluation is the same gate as above in pytest ergonomics - a plain AssertionError subclass, so it works in any runner with no plugin registration, and the verdict still lands in the dashboard’s CI gate history (caller="pytest"):

from agentx import AgentX
from agentx.testing import assert_evaluation

def test_support_agent_quality():
    client = AgentX.from_env()
    report = (
        client.evaluations
        .run(dataset_id=DATASET_ID, scorer_id=SCORER_ID, subject=SUBJECT)
        .execute(my_agent)
        .finalize()
    )
    assert_evaluation(report, min_rating=7.0, no_regression=True)

On failure the test output carries the per-check verdict plus the run id, so the red CI job links straight to the run in the dashboard:

  [FAIL] fail-under: Average rating 5.1 < floor 7

Self-Host CI Gate

Hosted CI/CD

CI/CD Evaluation

Prerequisites

High-level: `run_eval()`

Parameters

Return: `CIRunResult`

Low-level: step-by-step

Exception handling

Parallel question execution

Inspecting scores

Polling a run

GitHub Actions

Gating a Custom Agent Evaluation run

pytest-native assertions

​Prerequisites

​High-level: run_eval()

​Parameters

​Return: CIRunResult

​Low-level: step-by-step

​Exception handling

​Parallel question execution

​Inspecting scores

​Polling a run

​GitHub Actions

​Gating a Custom Agent Evaluation run

​pytest-native assertions

Prerequisites

High-level: `run_eval()`

Parameters

Return: `CIRunResult`

Low-level: step-by-step

Exception handling

Parallel question execution

Inspecting scores

Polling a run

GitHub Actions

Gating a Custom Agent Evaluation run

pytest-native assertions