CI/CD Evaluation

The CI/CD evaluation feature lets you run your agent against a test dataset in your CI pipeline and receive a binary PASS / FAIL gate result. If the gate fails, the pipeline exits with a non-zero code and blocks the merge or deploy.

Set AGENTX_EVAL_QUIET=1 in CI to silence the interactive progress UI (spinners, per-case lines) - gate verdicts, results, and errors still print, keeping a gated run’s log to a few lines.

Prerequisites

Create an evaluation dataset in AgentX with at least one question.
Enable CI/CD in the dataset settings and set a pass rate threshold.
Export AGENTX_API_KEY in your environment.

High-level: `run_eval()`

The easiest path: one call handles the entire lifecycle:

from agentx import AgentX, CIGateFailure

client = AgentX.from_env()

def my_agent(query: str) -> str:
    # your agent here
    return call_my_agent(query)

result = client.tracer.run_eval(
    dataset_id="6876ddd222bbb333ccc444ee",
    agent_fn=my_agent,
    agent_name="customer-support-agent",
    # Keys must be camelCase. The API's gitContext schema is strict
    # and silently drops unrecognized (e.g. snake_case) keys.
    git_context={
        "branch": "feat/new-retrieval",
        "commitSha": "a1b2c3d4e5f6",
    },
    fail_on_gate=True,   # raises CIGateFailure if gate is "fail"
)

print(f"Gate: {result.gate}  |  Pass rate: {result.pass_rate:.0%}")

Parameters

Parameter	Type	Default	Description
`dataset_id`	`str`	required	EvaluationSettings ID (must have `ci.enabled: true`)
`agent_fn`	`Callable[[str], str]`	required	Function that takes a query and returns a response
`agent_name`	`str`	none	Label for this agent on the AgentX platform
`pass_rate_threshold`	`float`	dataset default	Per-run override (0.0–1.0)
`git_context`	`dict`	none	Branch, commit SHA, PR number, etc.
`concurrency`	`int`	`1`	Max parallel question invocations
`fail_on_gate`	`bool`	`False`	Raise `CIGateFailure` if gate is `"fail"`
`timeout_per_question`	`float`	none	Seconds before a question times out

Return: `CIRunResult`

@dataclass
class CIRunResult:
    run_id: str
    gate: Literal["pass", "fail"]
    pass_rate: float                     # e.g. 0.875
    total_questions: int
    passed_questions: int
    scores: list[CIQuestionScore]
    violations: list[ThresholdViolation]
    finalized_at: str | None

Low-level: step-by-step

For custom orchestration: parallel execution, streaming, or external agents:

from agentx import AgentX
from agentx.tracing import CIRun, CIRunResult

client = AgentX.from_env()
tracer = client.tracer

# 1. Create the run, receive test cases
run: CIRun = tracer.create_ci_run(
    dataset_id="6876ddd222bbb333ccc444ee",
    agent_name="my-agent",
    git_context={"branch": "main", "commitSha": "abc1234"},
)

# 2. Run agent against each test case
for tc in run.test_cases:
    output = my_agent(tc.query or "")       # tc.query is None if exposeTestInputs is off
    score = tracer.submit_result(
        run.run_id,
        tc.index,
        output,
        latency_ms=350,
    )
    if score.gate_fired:
        print("failFast triggered, run already finalized as FAIL")
        break

# 3. Finalize and get the gate decision
result: CIRunResult = tracer.finalize_ci_run(run.run_id)
print(f"Gate: {result.gate}  ({result.passed_questions}/{result.total_questions} passed)")

Exception handling

from agentx import AgentX, CIGateFailure, CINotEnabled, DatasetNotFound

try:
    result = client.tracer.run_eval(
        dataset_id=dataset_id,
        agent_fn=my_agent,
        fail_on_gate=True,
    )
except DatasetNotFound:
    print("Dataset not found, check the dataset_id")
    sys.exit(1)
except CINotEnabled:
    print("CI not enabled on this dataset, enable it in dataset settings")
    sys.exit(1)
except CIGateFailure as e:
    result = e.result
    print(f"Gate FAILED, {result.pass_rate:.0%} passed")
    for v in result.violations:
        print(f"  Q{v.question_index}: {v.metric} was {v.actual:.2f}, threshold {v.threshold:.2f}")
    sys.exit(1)

Parallel question execution

Run multiple questions concurrently to speed up large datasets:

result = client.tracer.run_eval(
    dataset_id=dataset_id,
    agent_fn=my_agent,
    concurrency=4,       # run 4 questions in parallel
    fail_on_gate=True,
)

Inspecting scores

for score in result.scores:
    status = "✓" if score.passed else "✗"
    print(f"  [{status}] Q{score.question_index}: {score.rating}/10, {score.justification[:80]}")

Polling a run

If you submit results asynchronously, poll until finalized:

import time
from agentx.tracing import CIRunStatus

while True:
    status: CIRunStatus = tracer.get_ci_run(run_id)
    if status.status in ("completed", "failed"):
        break
    time.sleep(5)

print(f"Gate: {status.gate}")

GitHub Actions

See the GitHub Actions integration guide for a complete workflow template.

Python SDK

Integrations

CI/CD Evaluation

Prerequisites

High-level: `run_eval()`

Parameters

Return: `CIRunResult`

Low-level: step-by-step

Exception handling

Parallel question execution

Inspecting scores

Polling a run

GitHub Actions

​Prerequisites

​High-level: run_eval()

​Parameters

​Return: CIRunResult

​Low-level: step-by-step

​Exception handling

​Parallel question execution

​Inspecting scores

​Polling a run

​GitHub Actions

Prerequisites

High-level: `run_eval()`

Parameters

Return: `CIRunResult`

Low-level: step-by-step

Exception handling

Parallel question execution

Inspecting scores

Polling a run

GitHub Actions