curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"version": "feat/new-retrieval@a1b2c3d4"
},
"split": "smoke"
}'
import sys
report = (
client.evaluations.run(
"dS4tG7hNb2VxZ8kQ5wMyA",
subject={"kind": "custom_agent", "displayName": "Customer Support Bot",
"metadata": {"version": "feat/new-retrieval@a1b2c3d4"}},
split="smoke",
)
.execute(my_agent) # called once per case
.finalize()
)
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
CI Gate
Create CI Run
Start the evaluation run a CI pipeline will gate on
POST
/
api
/
v1
/
custom-agent-evaluations
/
runs
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"version": "feat/new-retrieval@a1b2c3d4"
},
"split": "smoke"
}'
import sys
report = (
client.evaluations.run(
"dS4tG7hNb2VxZ8kQ5wMyA",
subject={"kind": "custom_agent", "displayName": "Customer Support Bot",
"metadata": {"version": "feat/new-retrieval@a1b2c3d4"}},
split="smoke",
)
.execute(my_agent) # called once per case
.finalize()
)
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
A CI run is a standard evaluation run - this is the same endpoint as
Create Run, shown here in its CI framing. Create the
run at the start of the pipeline job, drive your agent through the dataset’s cases, submit and
finalize, then compute the gate.
Two fields matter most in CI:
evaluationSubject.version(orevaluationSubject.metadata.version): tag the run with the branch or commit under test, so version-to-version comparisons in the dashboard line up with your git history.split: run only the dataset cases tagged with a named split (e.g."smoke") for fast PR gates, keeping the full dataset for nightly runs.
Authentication
string
required
Project API key (use a CI secret, never source control).
Body
string
required
Dataset ID to evaluate against.
object
Metadata about the agent under test. In CI, include a
version tag:{
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"version": "feat/new-retrieval@a1b2c3d4"
}
string
Grade with a standalone grading config instead of the dataset’s own criteria.
string
default:"\"sdk\""
How the run was triggered.
string
Named case subset to cover (cases tagged via
main_question.splits). The caller filters
which cases it executes; the split is recorded on the run’s subject.Response
Returns201 Created with { runId, datasetId, status: "in_progress", smokeTestVariants } -
see Create Run for the full field reference. Read the
cases to execute from the dataset itself
(Get Test Cases).
Errors
| Status | Body | Meaning |
|---|---|---|
400 | { "error": "datasetId is required" } | Missing or non-string datasetId |
400 | { "error": "evaluationSettingsId must be a string" } | evaluationSettingsId present but not a string (or null) |
400 | { "error": "scorerGroupId must be a string" } | scorerGroupId present but not a string (or null) |
404 | { "error": "Scorer group not found" } | No scorer group with this id in the project |
404 | { "error": "Dataset not found" } | No dataset with this id in the project |
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"version": "feat/new-retrieval@a1b2c3d4"
},
"split": "smoke"
}'
import sys
report = (
client.evaluations.run(
"dS4tG7hNb2VxZ8kQ5wMyA",
subject={"kind": "custom_agent", "displayName": "Customer Support Bot",
"metadata": {"version": "feat/new-retrieval@a1b2c3d4"}},
split="smoke",
)
.execute(my_agent) # called once per case
.finalize()
)
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}