Skip to main content
Block merges and deploys when your agent’s pass rate drops below your configured threshold — using GitHub Actions as the runner.

​
Prerequisites

  1. A dataset with CI/CD enabled in AgentX settings.
  2. Your AGENTX_API_KEY stored as a GitHub Actions secret.
  3. Your AGENTX_DATASET_ID — visible in the dataset settings page URL.

​
Workflow template

Save as .github/workflows/eval.yml in your repository: 
name: AgentX Evaluation Gate

on:
  pull_request:
    branches: [main]

jobs:
  eval:
    name: Evaluate agent
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install agentx-python

      - name: Run AgentX evaluation gate
        env:
          AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
        run: |
          python -c "
          import os, sys
          from agentx import AgentX, CIGateFailure

          # Import your agent
          from your_package import my_agent

          client = AgentX.from_env()

          try:
              result = client.tracer.run_eval(
                  dataset_id=os.environ['AGENTX_DATASET_ID'],
                  agent_fn=my_agent,
                  agent_name='my-agent',
                  git_context={
                      'branch': os.environ.get('GITHUB_HEAD_REF', ''),
                      'commit_sha': os.environ.get('GITHUB_SHA', ''),
                      'pr_number': os.environ.get('GITHUB_REF', '').split('/')[-2],
                      'repo_url': 'https://github.com/' + os.environ.get('GITHUB_REPOSITORY', ''),
                      'triggered_by': 'github-actions',
                  },
                  fail_on_gate=True,
              )
              print(f'Gate PASSED — {result.pass_rate:.0%} passed')
          except CIGateFailure as e:
              print(f'Gate FAILED — {e.result.pass_rate:.0%} passed ({e.result.passed_questions}/{e.result.total_questions})')
              sys.exit(1)
          "
        env:
          AGENTX_DATASET_ID: ${{ secrets.AGENTX_DATASET_ID }}

​
Using a dedicated eval script

For larger test setups, use a separate script: 
# scripts/eval_gate.py
import os
import sys
from agentx import AgentX, CIGateFailure
from your_package import my_agent  # import your agent

client = AgentX.from_env()

try:
    result = client.tracer.run_eval(
        dataset_id=os.environ["AGENTX_DATASET_ID"],
        agent_fn=my_agent,
        agent_name="my-agent",
        concurrency=4,
        git_context={
            "branch": os.environ.get("GITHUB_HEAD_REF", ""),
            "commit_sha": os.environ.get("GITHUB_SHA", ""),
            "pr_number": os.environ.get("GITHUB_REF", "").split("/")[-2],
            "repo_url": "https://github.com/" + os.environ.get("GITHUB_REPOSITORY", ""),
            "triggered_by": "github-actions",
        },
        fail_on_gate=True,
    )
    print(f"Gate PASSED — {result.pass_rate:.0%} ({result.passed_questions}/{result.total_questions})")

except CIGateFailure as e:
    r = e.result
    print(f"Gate FAILED — {r.pass_rate:.0%} ({r.passed_questions}/{r.total_questions})")
    for v in r.violations:
        print(f"  Q{v.question_index}: {v.metric} = {v.actual:.2f} (threshold {v.threshold:.2f})")
    sys.exit(1)
In the workflow: 
- name: Run AgentX evaluation gate
  run: python scripts/eval_gate.py
  env:
    AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
    AGENTX_DATASET_ID: ${{ secrets.AGENTX_DATASET_ID }}

​
Adding required status checks

To block merges when the gate fails:
  1. Go to your GitHub repo → Settings → Branches
  2. Add a branch protection rule for main
  3. Enable Require status checks to pass before merging
  4. Add eval / Evaluate agent as a required check

​
Viewing results

Every CI run is recorded in the CI Runs tab in your AgentX workspace, showing:
  • Gate result (PASS / FAIL) badge
  • Per-question scores and justifications
  • Branch, commit SHA, and PR number
  • Threshold violations

​
Environment variables

VariableDescription
AGENTX_API_KEYYour workspace API key
AGENTX_DATASET_IDEvaluation dataset ID with CI enabled
GITHUB_HEAD_REFPR branch name (set by GitHub Actions)
GITHUB_SHACommit SHA (set by GitHub Actions)
GITHUB_REFFull ref string (set by GitHub Actions)
GITHUB_REPOSITORYorg/repo (set by GitHub Actions)