curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login screen." },
"timings": { "latencyMs": 1240, "inputTokens": 150, "outputTokens": 45 },
"traceId": "mJ3vQ8pTr2LqYw6bZk9Xd"
}
]
}'
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly identified the password reset flow and gave clear step-by-step instructions.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": 0.62,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"judgeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
{
"error": "Run is already in a terminal state"
}
Custom Evaluations
Submit Results
Submit a batch of agent outputs for synchronous judge scoring
POST
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
results
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login screen." },
"timings": { "latencyMs": 1240, "inputTokens": 150, "outputTokens": 45 },
"traceId": "mJ3vQ8pTr2LqYw6bZk9Xd"
}
]
}'
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly identified the password reset flow and gave clear step-by-step instructions.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": 0.62,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"judgeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
{
"error": "Run is already in a terminal state"
}
Submits up to 10 agent outputs for a run. Each new result is scored synchronously inside this
request - a judge LLM call per result, plus any enabled similarity metrics and code scorers -
so allow a generous client timeout for judged batches. Submissions are idempotent: resubmitting
an
idempotencyKey the run has already accepted returns the stored score without re-scoring.
Send results as you collect them rather than accumulating everything client-side: every batch
response carries
liveStatistics, so the run’s average rating is observable while the run is
still executing.Authentication
string
required
Project API key.
Path Parameters
string
required
Run ID returned by
POST /runs.Body
string
required
Identifier for this batch, stored on each result row.
array
required
Array of 1-10 result objects. Each result has:
Show Result object fields
Show Result object fields
string
required
Unique key for this result within the run. Prevents duplicate scoring on retry (the
engine also enforces uniqueness on
(runId, idempotencyKey) at insert, so a racing
duplicate counts as a duplicate, never an error). Recommended format:
{runId}:{caseId}:run-{runNumber}. A result without one is counted in
failedValidation and dropped.number
default:"0"
0-based index into the dataset’s
questions array. Selects the case’s
expectedResults, judgeGuideline, and other per-case grading inputs.number
default:"1"
Which repetition this is (1 through
numberOfRequests).string
Case identifier. Defaults to
case-{questionIndex}.object
Agent input:
{ "query": string }.object
Agent output:
{ "text": string }. Required unless error is set - a result with
neither is counted in failedValidation and dropped.object
Error that occurred:
{ "type": string, "message": string }. When set, the result is
stored with status "failed" and rating 0; no judge call is made.string
ID of a trace previously ingested via
Submit Trace (the Python SDK’s
client.tracer.trace(..., sync=True) returns it as span.trace_id). Linking a trace
unlocks trajectory-aware judging (the judge sees the actual tool-call path), the
trajectory and context match scorers, tool definitions for toolContext: "detailed"
configs, and the dashboard’s View Trace on the result.string | string[]
What the agent actually retrieved for this case - a string or a list of chunk strings.
Feeds
{context}-referencing judge prompts (the RAG metric pack) and the context match
scorer. When absent, the engine falls back to the linked trace’s recorded retrieval
spans, then the case’s pinned retrievalContext.object
Performance metrics:
{ "latencyMs": number, "inputTokens": number, "outputTokens": number }.boolean
default:"false"
Marks this row as a paraphrase variant from the run’s
smokeTestVariants. Variant rows
are excluded from every rating aggregate - liveStatistics, the run’s averageRating
and scorerBreakdown, case statistics, low-scoring cases, and CI-gate aggregates - but
still count toward the run’s resultCount.string
The variant phrasing that was actually asked, when
isSmokeTestVariant is true.The hosted platform’s
observableTrace and metadata result fields are ignored by the
self-host engine - they are accepted without error but not stored. Link a real trace via
traceId instead; it powers strictly more (trajectory judging, timeline, scorers).Response
string
Echoed back.
string
Echoed back.
number
New results scored and stored in this call.
number
Results skipped because their
idempotencyKey already existed on this run.number
Results dropped (missing
idempotencyKey, or both output.text and error absent).string
The run’s status, e.g.
"in_progress".array
Per-result scores, in submission order (validation-dropped results are omitted):
[{ idempotencyKey, rating, justification, status, vectorSimilarity, jaccardSimilarity, bleuScore, rougeScore, codeScorerResults, judgeScorerResults, deduped? }].
judgeScorerResults carries one { scorerId, name, rating, justification } entry per
additional judge scorer (or scorer-group judge member) on the run, null when there are
none.
status is "scored", "skipped" (judge could not score - the reason is in
justification), or "failed" (the result carried an error). Duplicates echo the stored
first-submission payload with deduped: true.object
Rating aggregate recomputed after this batch:
{ averageRating, minRating, maxRating, ratedCount, skippedCount, failedCount }. Available
on every batch response, no Analyze Run required;
the Python SDK exposes the same numbers as run.average_rating and friends.Errors
| Status | Body | Meaning |
|---|---|---|
400 | { "error": "batchId must be a non-empty string" } | Missing or invalid batchId |
400 | { "error": "results must be a non-empty array" } | Missing or empty results |
400 | { "error": "Batch size must not exceed 10" } | More than 10 results in one call |
400 | { "error": "Every results entry must be an object" } | A results element is null, an array, or not an object |
404 | { "error": "Run not found" } | No run with this id in the project |
409 | { "error": "Run is already in a terminal state" } | The run is "completed" or "failed"; no more results are accepted |
409 | { "error": "The scorer group grading this run no longer exists" } | The run’s scorer group was deleted mid-run; results are refused rather than silently regraded on the dataset’s own rubric |
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login screen." },
"timings": { "latencyMs": 1240, "inputTokens": 150, "outputTokens": 45 },
"traceId": "mJ3vQ8pTr2LqYw6bZk9Xd"
}
]
}'
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly identified the password reset flow and gave clear step-by-step instructions.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": 0.62,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"judgeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
{
"error": "Run is already in a terminal state"
}