Self-Host CI Gate

Run your golden dataset against the PR’s version of your agent, then gate the run: an absolute rating floor, a no-regression check against the dataset’s previous run, or both. The gate is a plain exit-code contract, so it drops into any CI system.

import sys
from agentx import AgentX

client = AgentX.from_env()

def my_agent(case):
    # your agent logic here
    return answer_string

report = (
    client.evaluations
    .run(dataset_id="<datasetId>", subject={"kind": "custom_agent", "framework": "raw_python"})
    .execute(my_agent)          # the PR's version of your agent
    .finalize()
)

gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
sys.exit(gate.exit_code)        # 0 = merge, 1 = block

The two checks:

fail_under - the run’s average judge rating must be at or above the floor. Works from the very first run.
no_regression - the average must not drop more than tolerance (default 0.5) below the dataset’s previous completed run. The tolerance exists because judge scores are noisy: an exact comparison would flake builds on variance, not regressions. Requires run history, so point CI at a persistent self-host instance rather than an ephemeral one.

gate() prints a per-check verdict into the CI log and returns a GateResult (passed, exit_code, average_rating, baseline_average, checks). The same check is a plain HTTP call if you’d rather script it raw:

curl "$AGENTX_API_BASE_URL/custom-agent-evaluations/runs/<runId>/gate?failUnder=7&noRegression=true" \
  -H "x-api-key: $AGENTX_API_KEY"

GitHub Actions

Pointing at a persistent self-host instance (recommended - run history enables no_regression):

name: Eval gate
on: pull_request

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install agentx-python
      - name: Run evals and gate
        env:
          AGENTX_API_BASE_URL: ${{ secrets.AGENTX_SELFHOST_URL }}   # https://agentx.internal:4700/api/v1
          AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}             # whatever your agent itself needs
        run: python ci/eval_gate.py   # the script above

No persistent instance reachable from CI? Launch an ephemeral engine inside the job and gate on the floor only (a fresh database has no baseline to regress against):

      - name: Start ephemeral engine
        run: |
          pip install agentx-python
          AGENTX_TRACE_EVAL_SKIP_WEB=1 nohup agentx-trace-eval > engine.log 2>&1 &
          sleep 5
      - name: Run evals and gate
        env:
          AGENTX_API_BASE_URL: http://localhost:4700/api/v1
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # The engine prints "Default project API key: agtx_..." at startup - store it as a
          # repository secret (there is no anonymous key endpoint).
          AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
        run: |
          python ci/eval_gate.py

What to gate on

Every recorded gate lands in the dashboard’s CI Gates history - the same list is readable from the SDK for scripting (e.g. a weekly “how often did gates fire” report):

for g in client.evaluations.list_gates():
    print(g["passed"], g["averageRating"], g["caller"], g["checks"])

The strongest gate is a dataset curated from production: real failures as regression cases mean “the PR doesn’t re-break what production already taught you.” Pair fail_under (an absolute quality bar) with no_regression (this PR made nothing worse) - they catch different failure modes.

Gate history in the dashboard

Every gate the SDK runs is recorded by default (record=True, with a caller label like "github-actions"), and the dashboard’s CI Gates tab (in the sidebar under Automations) lists that history newest-first: verdict, dataset, average vs baseline, which checks ran, who called. The same page has a preview (“would the latest run pass these thresholds?”) that is never recorded - so exploring thresholds can’t pollute the history real CI jobs write - plus the copy-paste setup snippets. Pass record=False to gate_run() for an unrecorded check from code.

Datasets from Production

Grow the golden dataset the gate runs against

Hosted CI/CD API

The hosted platform’s equivalent pipeline

Python SDK

Integrations

Self-Host CI Gate

GitHub Actions

What to gate on

Gate history in the dashboard

Datasets from Production

Hosted CI/CD API

​GitHub Actions

​What to gate on

​Gate history in the dashboard

Datasets from Production

Hosted CI/CD API

GitHub Actions

What to gate on

Gate history in the dashboard