curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
{
"error": "Run not found"
}
Custom Evaluations
Get Missing Results
Resume support: the idempotency keys a run has already accepted
GET
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
missing-results
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
{
"error": "Run not found"
}
Returns the idempotency keys this run has already stored, so a client resuming after a crash or
network failure can skip re-running (and re-paying for) already-submitted cases. The Python
SDK’s
execute() derives its keys deterministically as {runId}:{caseId}:run-{runNumber} and
uses exactly this endpoint to resume.
The engine cannot know your full case list, so it reports what was submitted, not what is
missing: the legacy
missing field is always an empty array. Compute missing cases
client-side as your own expected keys minus submittedKeys.Authentication
Path Parameters
Response
Always
[]. Legacy field kept for older SDK versions; the client computes missing cases
from submittedKeys.Errors
| Status | Body | Meaning |
|---|---|---|
404 | { "error": "Run not found" } | No run with this id in the project |
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
{
"error": "Run not found"
}

