curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-4-7" }] }'
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"error": "Run not found"
}
Custom Evaluations
Analyze Run
Run AI analysis over a run’s results and store the structured report
POST
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
analyze
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-4-7" }] }'
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"error": "Run not found"
}
Runs an LLM analysis over the run’s scored results and stores a structured narrative report:
overall summary, consistency, instruction adherence, response patterns, reasoning quality, tool
usage, and prioritized recommendations. Fetch the stored report with
Get Report.
The analysis prompt samples the worst 12 and best 5 rated results (all of them when the run
has 17 or fewer), and each sampled result is independently re-scored by up to 3 judge models to
produce per-item agreement evidence.
On the self-host engine this endpoint is synchronous: it returns only once the judges are
done, so the response status is already terminal (
"completed" or "failed"), and the first
GET /runs/:runId/analyze-status poll reads that same terminal status. The mode: "sync"
field says so on the wire. (Hosted AgentX queues a durable job and returns "pending"
instead; the SDK’s poll loop handles both.) Re-running analysis overwrites the stored row.Authentication
string
required
Project API key.
Path Parameters
string
required
Run ID. Analyze after finalizing, so the analysis
covers every result.
Body
array
Judge models to score the evidence sample with, up to 3:
[{ "model": string }]. Model ids
come from the model catalog. The first entry also
writes the narrative. Omit to use the default judge model alone.string
default:"\"balanced\""
"balanced" or "quality_first". Echoed back; recorded with the analysis request.Response
string
The run ID (analysis is keyed by run).
string
Same value as
evaluationId, kept for SDK compatibility with the hosted job queue.string
"completed", or "failed" (for example, when the run has no scored results yet, or the
judge model returned no analysis - the failure reason is readable from
GET /runs/:runId/analyze-status).string
Always
"sync" on self-host.string
Echoed back.
Errors
| Status | Body | Meaning |
|---|---|---|
404 | { "error": "Run not found" } | No run with this id in the project |
Checking status
GET /api/v1/custom-agent-evaluations/runs/{runId}/analyze-status returns
{ evaluationId, jobId, status, progress, failureReason, warnings, cost, etaUpdatedAt, overflowStats }.
status is "not_started" before any analysis, otherwise the stored row’s terminal
"completed" or "failed"; a failure carries
failureReason: { code: "ANALYSIS_FAILED", message, retryable: true }. The progress/cost/
overflow fields exist for wire compatibility with the hosted job queue and are static on
self-host.
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-4-7" }] }'
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
{
"error": "Run not found"
}