Self-Host CI Gate

Run your golden dataset against the PR’s version of your agent, then gate the run: an absolute rating floor, a no-regression check against the dataset’s previous run, or both. The gate is a plain exit-code contract, so it drops into any CI system.

# ci/eval_gate.py
import sys
from agentx import AgentX

client = AgentX.from_env()  # reads AGENTX_API_BASE_URL and AGENTX_API_KEY

def my_agent(case):
    # your agent logic here
    return answer_string

run = (
    client.evaluations
    .run(dataset_id="<datasetId>", subject={"kind": "custom_agent", "framework": "raw_python"})
    .execute(my_agent)          # the PR's version of your agent
    .finalize()
)

gate = run.gate(fail_under=7, no_regression=True, caller="github-actions")
sys.exit(gate.exit_code)        # 0 = merge, 1 = block

The two checks:

fail_under - the run’s average judge rating must be at or above the floor. Works from the very first run.
no_regression - the average must not drop more than tolerance (default 0.5) below the dataset’s previous completed run. The tolerance exists because judge scores are noisy: an exact comparison would flake builds on variance, not regressions. Requires run history, so point CI at a persistent self-host instance rather than an ephemeral one.

The baseline must have measured the same thing as the gated run: only completed runs with the same evaluationSettingsId, scorerGroupId, and split qualify, single-trace evaluations are excluded, and only the 5 newest candidates are considered. With no matching run the check passes explicitly (“no previous completed run - nothing to regress against”). That means changing the grading config, the scorer group, or the split silently resets the baseline and no_regression passes on the next run - by design, but know it when you rework grading. On a multi-judge run, scorer gates a named additional judge scorer’s average instead of the primary rating - run.gate(fail_under=8, scorer="Safety") fails the build when Safety is low even if the blended average looks fine. The name (or id) must match an additional judge scorer on the run; an unknown name is a 400, and deterministic scorer-group members (pattern/code kinds) have no per-run average to gate. gate() prints a per-check verdict into the CI log and returns a GateResult (passed, exit_code, average_rating, baseline_average, checks). The same check is a plain HTTP call if you’d rather script it raw - add record=true (what the SDK sends by default) so the verdict lands in the dashboard’s gate history, with an optional caller label. Note that with record=True the SDK deliberately disables transport retry: despite the GET verb, recording is a server-side write, and a timeout after the row was stored would be retried into a duplicate verdict:

curl "$AGENTX_API_BASE_URL/custom-agent-evaluations/runs/<runId>/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
  -H "x-api-key: $AGENTX_API_KEY"

GitHub Actions

Pointing at a persistent self-host instance (recommended - run history enables no_regression):

name: Eval gate
on: pull_request

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install agentx-python
      - name: Run evals and gate
        env:
          AGENTX_API_BASE_URL: ${{ secrets.AGENTX_SELFHOST_URL }}   # https://agentx.internal:4700/api/v1
          AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}             # whatever your agent itself needs
        run: python ci/eval_gate.py   # the script above

No persistent instance reachable from CI? Launch an ephemeral engine inside the job and gate on the floor only (a fresh database has no baseline to regress against):

      - name: Start ephemeral engine
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}   # the engine's judge needs a provider key
        run: |
          pip install agentx-python
          AGENTX_TRACE_EVAL_SKIP_WEB=1 nohup agentx-trace-eval > engine.log 2>&1 &
          # First run downloads the engine release, so wait on /health rather than sleeping
          timeout 120 bash -c 'until curl -fsS http://localhost:4700/health; do sleep 2; done'
      - name: Run evals and gate
        env:
          AGENTX_API_BASE_URL: http://localhost:4700/api/v1
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}   # whatever your agent itself needs
        run: |
          # In the default auth-disabled mode the engine hands out the Default project's key
          export AGENTX_API_KEY=$(curl -fsS http://localhost:4700/api/v1/auth/config | python3 -c "import json,sys; print(json.load(sys.stdin)['apiKey'])")
          python ci/eval_gate.py

What to gate on

Every recorded gate lands in the dashboard’s CI Gates history - the same list is readable from the SDK for scripting (e.g. a weekly “how often did gates fire” report):

for g in client.evaluations.list_gates():
    print(g["passed"], g["averageRating"], g["caller"], g["checks"])

The strongest gate is a dataset curated from production: real failures as regression cases mean “the PR doesn’t re-break what production already taught you.” Pair fail_under (an absolute quality bar) with no_regression (this PR made nothing worse) - they catch different failure modes.

Gate history in the dashboard

Every gate the SDK runs is recorded by default (record=True, with a caller label like "github-actions"), and the dashboard’s CI Gates tab (in the sidebar under Automations) lists that history newest-first: verdict, dataset, average vs baseline, which checks ran, who called. The same page has a preview (“would the latest run pass these thresholds?”) that is never recorded - so exploring thresholds can’t pollute the history real CI jobs write - plus the copy-paste setup snippets. Pass record=False to gate_run() for an unrecorded check from code.

Datasets from Production

Grow the golden dataset the gate runs against

Hosted CI/CD API

The hosted platform’s equivalent pipeline

TypeScript CI SDK

The same gate from a TypeScript pipeline, via @agentx/eval

​GitHub Actions

​What to gate on

​Gate history in the dashboard