> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-Host CI Gate

> Block a merge when your agent's eval score drops - an exit-code contract over a normal eval run

Run your golden dataset against the PR's version of your agent, then **gate** the run: an absolute rating floor, a no-regression check against the dataset's previous run, or both. The gate is a plain exit-code contract, so it drops into any CI system.

```python theme={null}
import sys
from agentx import AgentX

client = AgentX.from_env()

def my_agent(case):
    # your agent logic here
    return answer_string

report = (
    client.evaluations
    .run(dataset_id="<datasetId>", subject={"kind": "custom_agent", "framework": "raw_python"})
    .execute(my_agent)          # the PR's version of your agent
    .finalize()
)

gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
sys.exit(gate.exit_code)        # 0 = merge, 1 = block
```

The two checks:

* **`fail_under`** - the run's average judge rating must be at or above the floor. Works from the very first run.
* **`no_regression`** - the average must not drop more than `tolerance` (default `0.5`) below the dataset's previous completed run. The tolerance exists because judge scores are noisy: an exact comparison would flake builds on variance, not regressions. Requires run history, so point CI at a **persistent** self-host instance rather than an ephemeral one.

`gate()` prints a per-check verdict into the CI log and returns a `GateResult` (`passed`, `exit_code`, `average_rating`, `baseline_average`, `checks`). The same check is a plain HTTP call if you'd rather script it raw:

```bash theme={null}
curl "$AGENTX_API_BASE_URL/custom-agent-evaluations/runs/<runId>/gate?failUnder=7&noRegression=true" \
  -H "x-api-key: $AGENTX_API_KEY"
```

## GitHub Actions

Pointing at a persistent self-host instance (recommended - run history enables `no_regression`):

```yaml theme={null}
name: Eval gate
on: pull_request

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install agentx-python
      - name: Run evals and gate
        env:
          AGENTX_API_BASE_URL: ${{ secrets.AGENTX_SELFHOST_URL }}   # https://agentx.internal:4700/api/v1
          AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}             # whatever your agent itself needs
        run: python ci/eval_gate.py   # the script above
```

No persistent instance reachable from CI? Launch an ephemeral engine inside the job and gate on the floor only (a fresh database has no baseline to regress against):

```yaml theme={null}
      - name: Start ephemeral engine
        run: |
          pip install agentx-python
          AGENTX_TRACE_EVAL_SKIP_WEB=1 nohup agentx-trace-eval > engine.log 2>&1 &
          sleep 5
      - name: Run evals and gate
        env:
          AGENTX_API_BASE_URL: http://localhost:4700/api/v1
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # The engine prints "Default project API key: agtx_..." at startup - store it as a
          # repository secret (there is no anonymous key endpoint).
          AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
        run: |
          python ci/eval_gate.py
```

## What to gate on

Every recorded gate lands in the dashboard's **CI Gates** history - the same list is
readable from the SDK for scripting (e.g. a weekly "how often did gates fire" report):

```python theme={null}
for g in client.evaluations.list_gates():
    print(g["passed"], g["averageRating"], g["caller"], g["checks"])
```

The strongest gate is a dataset [curated from production](/evaluation/datasets-from-production): real failures as regression cases mean "the PR doesn't re-break what production already taught you." Pair `fail_under` (an absolute quality bar) with `no_regression` (this PR made nothing worse) - they catch different failure modes.

## Gate history in the dashboard

Every gate the SDK runs is **recorded by default** (`record=True`, with a `caller` label like `"github-actions"`), and the dashboard's **CI Gates** tab (in the sidebar under Automations) lists that history newest-first: verdict, dataset, average vs baseline, which checks ran, who called. The same page has a **preview** ("would the latest run pass these thresholds?") that is never recorded - so exploring thresholds can't pollute the history real CI jobs write - plus the copy-paste setup snippets. Pass `record=False` to `gate_run()` for an unrecorded check from code.

<CardGroup cols={2}>
  <Card title="Datasets from Production" icon="database" href="/evaluation/datasets-from-production">
    Grow the golden dataset the gate runs against
  </Card>

  <Card title="Hosted CI/CD API" icon="github" href="/sdk/ci-cd">
    The hosted platform's equivalent pipeline
  </Card>
</CardGroup>