> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# CI/CD Evaluation

> Gate agent releases with eval test sets

The CI/CD evaluation feature lets you run your agent against a test dataset in your CI pipeline and receive a binary **PASS / FAIL** gate result. If the gate fails, the pipeline exits with a non-zero code and blocks the merge or deploy.

## Prerequisites

1. Create an evaluation dataset in AgentX with at least one question.
2. Enable **CI/CD** in the dataset settings and set a pass rate threshold.
3. Export `AGENTX_API_KEY` in your environment.

## High-level: `run_eval()`

The easiest path — one call handles the entire lifecycle:

```python theme={null}
from agentx import AgentX, CIGateFailure

client = AgentX.from_env()

def my_agent(query: str) -> str:
    # your agent here
    return call_my_agent(query)

result = client.tracer.run_eval(
    dataset_id="6876ddd222bbb333ccc444ee",
    agent_fn=my_agent,
    agent_name="customer-support-agent",
    git_context={
        "branch": "feat/new-retrieval",
        "commit_sha": "a1b2c3d4e5f6",
    },
    fail_on_gate=True,   # raises CIGateFailure if gate is "fail"
)

print(f"Gate: {result.gate}  |  Pass rate: {result.pass_rate:.0%}")
```

### Parameters

| Parameter              | Type                   | Default         | Description                                          |
| ---------------------- | ---------------------- | --------------- | ---------------------------------------------------- |
| `dataset_id`           | `str`                  | required        | EvaluationSettings ID (must have `ci.enabled: true`) |
| `agent_fn`             | `Callable[[str], str]` | required        | Function that takes a query and returns a response   |
| `agent_name`           | `str`                  | —               | Label for this agent on the AgentX platform          |
| `pass_rate_threshold`  | `float`                | dataset default | Per-run override (0.0–1.0)                           |
| `git_context`          | `dict`                 | —               | Branch, commit SHA, PR number, etc.                  |
| `concurrency`          | `int`                  | `1`             | Max parallel question invocations                    |
| `fail_on_gate`         | `bool`                 | `False`         | Raise `CIGateFailure` if gate is `"fail"`            |
| `timeout_per_question` | `float`                | —               | Seconds before a question times out                  |

### Return: `CIRunResult`

```python theme={null}
@dataclass
class CIRunResult:
    run_id: str
    gate: Literal["pass", "fail"]
    pass_rate: float                     # e.g. 0.875
    total_questions: int
    passed_questions: int
    scores: list[CIQuestionScore]
    violations: list[ThresholdViolation]
    finalized_at: str | None
```

## Low-level: step-by-step

For custom orchestration — parallel execution, streaming, or external agents:

```python theme={null}
from agentx import AgentX
from agentx.tracing import CIRun, CIRunResult

client = AgentX.from_env()
tracer = client.tracer

# 1. Create the run, receive test cases
run: CIRun = tracer.create_ci_run(
    dataset_id="6876ddd222bbb333ccc444ee",
    agent_name="my-agent",
    git_context={"branch": "main", "commit_sha": "abc1234"},
)

# 2. Run agent against each test case
for tc in run.test_cases:
    output = my_agent(tc.query or "")       # tc.query is None if exposeTestInputs is off
    score = tracer.submit_result(
        run.run_id,
        tc.index,
        output,
        latency_ms=350,
    )
    if score.gate_fired:
        print("failFast triggered — run already finalized as FAIL")
        break

# 3. Finalize and get the gate decision
result: CIRunResult = tracer.finalize_ci_run(run.run_id)
print(f"Gate: {result.gate}  ({result.passed_questions}/{result.total_questions} passed)")
```

## Exception handling

```python theme={null}
from agentx import AgentX, CIGateFailure, CINotEnabled, DatasetNotFound

try:
    result = client.tracer.run_eval(
        dataset_id=dataset_id,
        agent_fn=my_agent,
        fail_on_gate=True,
    )
except DatasetNotFound:
    print("Dataset not found — check the dataset_id")
    sys.exit(1)
except CINotEnabled:
    print("CI not enabled on this dataset — enable it in dataset settings")
    sys.exit(1)
except CIGateFailure as e:
    result = e.result
    print(f"Gate FAILED — {result.pass_rate:.0%} passed")
    for v in result.violations:
        print(f"  Q{v.question_index}: {v.metric} was {v.actual:.2f}, threshold {v.threshold:.2f}")
    sys.exit(1)
```

## Parallel question execution

Run multiple questions concurrently to speed up large datasets:

```python theme={null}
result = client.tracer.run_eval(
    dataset_id=dataset_id,
    agent_fn=my_agent,
    concurrency=4,       # run 4 questions in parallel
    fail_on_gate=True,
)
```

## Inspecting scores

```python theme={null}
for score in result.scores:
    status = "✓" if score.passed else "✗"
    print(f"  [{status}] Q{score.question_index}: {score.rating}/5 — {score.justification[:80]}")
```

## Polling a run

If you submit results asynchronously, poll until finalized:

```python theme={null}
import time
from agentx.tracing import CIRunStatus

while True:
    status: CIRunStatus = tracer.get_ci_run(run_id)
    if status.status in ("completed", "failed"):
        break
    time.sleep(5)

print(f"Gate: {status.gate}")
```

## GitHub Actions

See the [GitHub Actions integration guide](/integrations/github-actions) for a complete workflow template.