curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-5" }] }'
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"error": "Run not found"
}
Custom Evaluations
Analyze Run
Run AI analysis over a run’s results and store the structured report
POST
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
analyze
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-5" }] }'
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"error": "Run not found"
}
Runs an LLM analysis over the run’s scored results and stores a structured narrative report:
overall summary, consistency, instruction adherence, response patterns, reasoning quality, tool
usage, and prioritized recommendations. Fetch the stored report with
Get Report.
The analysis prompt samples the worst 12 and best 5 rated results (all of them when the run
has 17 or fewer), and each sampled result is independently re-scored by up to 3 judge models to
produce per-item agreement evidence.
On the self-host engine this endpoint is synchronous: it returns only once the judges are
done, so the response status is already terminal (
"completed" or "failed"), and the first
GET /runs/:runId/analyze-status poll reads that same terminal status. The mode: "sync"
field says so on the wire. (Hosted AgentX queues a durable job and returns "pending"
instead; the SDK’s poll loop handles both.) Re-running analysis overwrites the stored row.Authentication
string
required
Project API key.
Path Parameters
string
required
Run ID. Analyze after finalizing, so the analysis
covers every result.
Body
array
Judge models to score the evidence sample with, up to 3:
[{ "model": string }]. Model ids
come from the model catalog. Duplicate models are
deduplicated before the 3-model cap is applied. The first entry also writes the narrative.
Omit to use the default judge model alone.string
default:"\"balanced\""
"balanced" or "quality_first". Echoed back in the response; not persisted with the stored
report.A second analyze request for the same run while one is already in flight joins the running
analysis and returns its result instead of starting a duplicate - so an immediate retry does
not double your judge spend or re-run the analysis.
Response
string
The run ID (analysis is keyed by run).
string
Same value as
evaluationId, kept for SDK compatibility with the hosted job queue.string
"completed", or "failed" (for example, when the run has no scored results yet, or the
judge model returned no analysis - the failure reason is readable from
GET /runs/:runId/analyze-status).string
Always
"sync" on self-host.string
Echoed back.
Errors
| Status | Body | Meaning |
|---|---|---|
404 | { "error": "Run not found" } | No run with this id in the project |
Checking status
GET /api/v1/custom-agent-evaluations/runs/{runId}/analyze-status returns
{ evaluationId, jobId, status, progress, failureReason, warnings, cost, etaUpdatedAt, overflowStats }.
status is "not_started" before any analysis, "running" while one is in flight for the
run (a concurrent or retried analyze), otherwise the stored row’s terminal "completed" or
"failed"; a failure carries failureReason: { code: "ANALYSIS_FAILED", message, retryable: true }. Poll loops must treat any non-terminal value as “keep waiting”, not just
"not_started". All three shapes carry the same keys; running and terminal rows carry
jobId set to the run ID, and only not_started carries jobId: null. On terminal rows the
progress/cost/overflow fields exist for wire compatibility with the hosted job queue and are
static on self-host. An unknown run returns 404 { "error": "Run not found" }.
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-5" }] }'
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"error": "Run not found"
}