curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
import sys
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
from agentx.testing import assert_evaluation
def test_support_agent_quality():
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
# Same gate in pytest ergonomics: raises EvaluationAssertionError on failure
# (recorded in gate history with caller="pytest").
assert_evaluation(report, min_rating=7, no_regression=True)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"gatedScorer": null,
"averageRating": 7.8,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": true,
"threshold": 7,
"actual": 7.8,
"detail": "Average rating 7.8 >= floor 7"
},
{
"check": "no-regression",
"passed": true,
"threshold": 7.5,
"actual": 7.8,
"detail": "Average rating 7.8 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": true
}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"gatedScorer": null,
"averageRating": 5.2,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": false,
"threshold": 7,
"actual": 5.2,
"detail": "Average rating 5.2 < floor 7"
},
{
"check": "no-regression",
"passed": false,
"threshold": 7.5,
"actual": 5.2,
"detail": "Average rating 5.2 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": false
}
{
"error": "At least one check is required: failUnder=<0-10> and/or noRegression=true"
}
CI Gate
Get CI Gate
Compute the pass/fail verdict for a finalized run
GET
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
gate
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
import sys
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
from agentx.testing import assert_evaluation
def test_support_agent_quality():
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
# Same gate in pytest ergonomics: raises EvaluationAssertionError on failure
# (recorded in gate history with caller="pytest").
assert_evaluation(report, min_rating=7, no_regression=True)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"gatedScorer": null,
"averageRating": 7.8,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": true,
"threshold": 7,
"actual": 7.8,
"detail": "Average rating 7.8 >= floor 7"
},
{
"check": "no-regression",
"passed": true,
"threshold": 7.5,
"actual": 7.8,
"detail": "Average rating 7.8 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": true
}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"gatedScorer": null,
"averageRating": 5.2,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": false,
"threshold": 7,
"actual": 5.2,
"detail": "Average rating 5.2 < floor 7"
},
{
"check": "no-regression",
"passed": false,
"threshold": 7.5,
"actual": 5.2,
"detail": "Average rating 5.2 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": false
}
{
"error": "At least one check is required: failUnder=<0-10> and/or noRegression=true"
}
Computes the CI gate for a run: an absolute rating floor, a no-regression check against the
dataset’s previous completed run, or both. The verdict is computed fresh from the run’s stored
ratings on every call, so re-running a failed CI job re-evaluates against current state. Exit
your pipeline non-zero when
passed is false.
To poll a run’s progress or read its per-result scores, use
Get Run instead; this endpoint is the verdict.
Authentication
string
required
Project API key.
Path Parameters
string
required
Run ID. Finalize the run first so the gate covers every
result - the endpoint does not enforce this, and gating an
in_progress run scores whatever
has been submitted so far.Query Parameters
At least one offailUnder and noRegression is required.
number
Absolute floor, 0-10: the check fails when the run’s average rating is below this value (or
when the run has no rated results).
boolean
true (or 1): fail when the average dropped more than tolerance below the baseline run’s
average. The baseline is the dataset’s most recent earlier completed rated run with the
same grading identity: same evaluationSettingsId, same scorerGroupId, and same
evaluationSubject.split - a run graded by a different config, a different scorer group, or
on a different split is never your baseline, and only the 5 most recent matching candidates
are considered. Passes explicitly when no baseline exists. Trace evaluations
(runSource: "trace-eval") are never used as the baseline.number
default:"0.5"
Allowed drop for the no-regression check. Judge scores are noisy; an exact comparison would
flake builds on variance rather than regressions.
boolean
default:"false"
true (or 1): persist this verdict into the project’s gate history (the dashboard’s CI
page). A recorded failed gate also fires the webhook channels configured on the project’s
monitoring profiles. The Python SDK’s report.gate() records by default; ad hoc previews
should omit it.string
Label stored with a recorded verdict (max 60 characters), e.g.
"github-actions" or
"pytest".string
Gate on a named additional judge scorer instead of the primary (multi-judge runs, self-host):
pass the scorer’s id or name, e.g.
scorer=Safety. failUnder then checks that scorer’s own
average, and noRegression compares against the previous run’s average for the same scorer.
The response echoes it back as gatedScorer: { id, name } (null when gating the primary).
An unknown id or name is a 400, never a silently-passing gate.Response
string
Run ID.
string
Dataset the run scored against.
number | null
The run’s average rating, rounded to 2 decimals.
null when nothing was rated.number
Total submitted results.
string | null
The previous completed rated run used as the no-regression baseline, or
null (check not
requested, or no run with the same grading identity exists - see noRegression above; a
null here after switching grading config, scorer group, or split is expected).number | null
The baseline run’s average rating.
object | null
{ id, name } of the additional scorer being gated when ?scorer= was passed; null when
the gate ran against the primary scorer. When set, averageRating, baselineAverage, and
every check use that scorer’s own per-result verdicts.array
One entry per requested check:
[{ "check": "fail-under" | "no-regression", "passed": boolean, "threshold": number | null, "actual": number | null, "detail": string }].
detail is a CI-log-friendly sentence explaining the outcome.boolean
true only when every requested check passed. This is the field to exit on.Errors
| Status | Body | Meaning |
|---|---|---|
400 | { "error": "At least one check is required: failUnder=<0-10> and/or noRegression=true" } | Neither check requested |
400 | { "error": "failUnder must be a number" } | Non-numeric failUnder |
400 | { "error": "tolerance must be a non-negative number" } | Non-numeric or negative tolerance |
400 | { "error": "No additional scorer \"<name>\" on this run - gate names must match an additional scorer's id or name" } | ?scorer= did not match any additional scorer on the run |
404 | { "error": "Run not found" } | No run with this id in the project |
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
import sys
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
from agentx.testing import assert_evaluation
def test_support_agent_quality():
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
# Same gate in pytest ergonomics: raises EvaluationAssertionError on failure
# (recorded in gate history with caller="pytest").
assert_evaluation(report, min_rating=7, no_regression=True)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"gatedScorer": null,
"averageRating": 7.8,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": true,
"threshold": 7,
"actual": 7.8,
"detail": "Average rating 7.8 >= floor 7"
},
{
"check": "no-regression",
"passed": true,
"threshold": 7.5,
"actual": 7.8,
"detail": "Average rating 7.8 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": true
}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"gatedScorer": null,
"averageRating": 5.2,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": false,
"threshold": 7,
"actual": 5.2,
"detail": "Average rating 5.2 < floor 7"
},
{
"check": "no-regression",
"passed": false,
"threshold": 7.5,
"actual": 5.2,
"detail": "Average rating 5.2 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": false
}
{
"error": "At least one check is required: failUnder=<0-10> and/or noRegression=true"
}