Webhooks

Configure a webhook URL on a dataset to have AgentX POST the gate result to your server immediately after a hosted CI/CD run is finalized. (The self-host gate returns its verdict synchronously instead - gate history is queryable, no webhook involved.)

Configuration

Set webhookUrl on the dataset in Manage → Datasets → [dataset] → CI/CD, or include it when creating a dataset via the API:

{
  "ci": {
    "enabled": true,
    "webhookUrl": "https://your-server.com/hooks/agentx"
  }
}

Delivery

Sent as a single POST request with Content-Type: application/json
Fire-and-forget: AgentX does not retry on failure or timeout
Delivery is best-effort; build idempotency into your handler using run_id

When the dataset’s ci.failFast trips, the run is marked failed at submit time and finalize is never reached - so no webhook fires at all for fail-fast runs. A listener that blocks waiting on ci_run.finalized will hang; treat the missing webhook as a possible fail-fast failure and poll the run via the API if you need a definitive verdict.

Payload

{
  "event": "ci_run.finalized",
  "run_id": "6876ccc111aaa222bbb333dd",
  "dataset_id": "6876ddd222bbb333ccc444ee",
  "gate": "pass",
  "pass_rate": 0.875,
  "git_context": {
    "branch": "feat/new-retrieval",
    "commitSha": "a1b2c3d4e5f6",
    "prNumber": 42,
    "repoUrl": "https://github.com/org/repo",
    "triggeredBy": "github-actions"
  },
  "scores": [
    {
      "question_index": 0,
      "rating": 8,
      "justification": "The agent correctly described the password reset flow.",
      "passed": true,
      "input": "How do I reset my password?",
      "output": "Click Forgot Password on the login screen."
    },
    {
      "question_index": 1,
      "rating": 2,
      "justification": "The agent gave an incorrect billing answer.",
      "passed": false,
      "input": "Why was I charged twice?",
      "output": "..."
    }
  ],
  "finalized_at": "2026-07-06T11:00:00.000Z"
}

Fields

Field	Type	Description
`event`	string	Always `"ci_run.finalized"`
`run_id`	string	CI run ID
`dataset_id`	string	Dataset (EvaluationSettings) ID
`gate`	`"pass"` \| `"fail"`	Gate result
`pass_rate`	number	Fraction of questions that passed (0.0-1.0)
`git_context`	object \| null	Branch, commit SHA, PR number, etc. (from `create_ci_run`)
`scores`	array	Per-question scores (see below)
`finalized_at`	string	ISO 8601 finalization timestamp

`scores[n]` fields

Field	Type	Description
`question_index`	number	0-based question index
`rating`	number	LLM score, 0-10 (an unscored result is coerced to `0`)
`justification`	string	LLM explanation
`passed`	boolean	Whether the question passed all threshold gates
`input`	string \| null	Question text - if the submitter omitted it, the server backfills the dataset’s question text at submit time, so it is populated even when `exposeTestInputs` is false
`output`	string \| null	Agent’s response text

Example receiver

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/hooks/agentx", methods=["POST"])
def handle_agentx_webhook():
    payload = request.get_json()

    if payload.get("event") != "ci_run.finalized":
        return jsonify({"ok": True})

    run_id = payload["run_id"]
    gate = payload["gate"]
    pass_rate = payload["pass_rate"]
    branch = (payload.get("git_context") or {}).get("branch", "unknown")

    print(f"[{branch}] CI run {run_id}: {gate.upper()} ({pass_rate:.0%})")

    if gate == "fail":
        # notify Slack, update GitHub status, etc.
        notify_team(run_id, payload["scores"])

    return jsonify({"ok": True}), 200

import express from "express";

const app = express();
app.use(express.json());

app.post("/hooks/agentx", (req, res) => {
  const { event, run_id, gate, pass_rate, git_context, scores } = req.body;

  if (event !== "ci_run.finalized") return res.json({ ok: true });

  const branch = git_context?.branch ?? "unknown";
  console.log(`[${branch}] CI run ${run_id}: ${gate.toUpperCase()} (${Math.round(pass_rate * 100)}%)`);

  if (gate === "fail") {
    const failing = scores.filter((s: any) => !s.passed);
    console.log(`Failing questions: ${failing.map((s: any) => s.question_index).join(", ")}`);
  }

  res.json({ ok: true });
});

Verifying delivery

AgentX does not sign webhook payloads currently, so treat the webhook as a notification only - anything decision-worthy (gating a deploy, paging someone) should be verified by using the run_id to fetch the authoritative result back from the API:

import requests

@app.route("/hooks/agentx", methods=["POST"])
def handle():
    payload = request.get_json()
    run_id = payload["run_id"]

    # Verify with the API
    result = requests.get(
        f"https://api.agentx.so/api/v1/ingest/ci-runs/{run_id}",
        headers={"x-api-key": API_KEY},
    ).json()

    assert result["gate"] == payload["gate"]
    return "", 200

Self-Host CI Gate

Hosted CI/CD

Configuration

Delivery

Payload

Fields

`scores[n]` fields

Example receiver

Verifying delivery

​Configuration

​Delivery

​Payload

​Fields

​scores[n] fields

​Example receiver

​Verifying delivery

Configuration

Delivery

Payload

Fields

`scores[n]` fields

Example receiver

Verifying delivery