curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "ci-batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login page to reset your password." },
"timings": { "latencyMs": 1340 }
}
]
}'
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "ci-batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly described the password reset flow.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": null,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"judgeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
{
"error": "Run is already in a terminal state"
}
CI Gate
Submit Result
Submit and score CI case results as the pipeline produces them
POST
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
results
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "ci-batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login page to reset your password." },
"timings": { "latencyMs": 1340 }
}
]
}'
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "ci-batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly described the password reset flow.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": null,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"judgeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
{
"error": "Run is already in a terminal state"
}
Submits a batch of up to 10 case results for the CI run; each new result is judge-scored
synchronously inside the request. This is the same endpoint as
Submit Results - see that page for the full
result-object reference. This page covers the CI angle.
CI-relevant behavior
- Idempotency makes retries safe. Give every result an
idempotencyKey({runId}:{caseId}:run-{runNumber}); a retried or duplicated batch returns the stored scores instead of re-scoring, so a flaky network step never double-bills judge calls. On job restart, fetch the already-submitted keys from Get Missing Results and skip them. - Failures are results too. A case where your agent threw should be submitted with
error: { "type", "message" }- it scores 0 and drags the average down honestly, instead of silently shrinking the run. - Watch the average as you go. Every batch response carries
liveStatistics; a pipeline can log progress or bail out early when the average is already hopeless. There is no server-side fail-fast: stopping early is the client’s decision, and the gate computes over whatever was submitted. - Generous timeouts. Scoring is a judge LLM call per result; the Python SDK uses a long per-batch timeout and disables transport retries for exactly this call.
Errors
| Status | Body | Meaning |
|---|---|---|
400 | { "error": "batchId must be a non-empty string" } | Missing or invalid batchId |
400 | { "error": "results must be a non-empty array" } | Missing or empty results |
400 | { "error": "Batch size must not exceed 10" } | More than 10 results in one call |
400 | { "error": "Every results entry must be an object" } | A results element is null, an array, or not an object |
404 | { "error": "Run not found" } | No run with this id in the project |
409 | { "error": "Run is already in a terminal state" } | The run was already finalized or failed |
409 | { "error": "The scorer group grading this run no longer exists" } | The run’s scorer group was deleted mid-run; results are refused rather than silently regraded |
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "ci-batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login page to reset your password." },
"timings": { "latencyMs": 1340 }
}
]
}'
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "ci-batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly described the password reset flow.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": null,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"judgeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
{
"error": "Run is already in a terminal state"
}