curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
{
"error": "Run not found"
}
Custom Evaluations
Get Missing Results
Resume support: the idempotency keys a run has already accepted
GET
/
api
/
v1
/
custom-agent-evaluations
/
runs
/
{runId}
/
missing-results
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
{
"error": "Run not found"
}
Returns the idempotency keys this run has already stored, so a client resuming after a crash or
network failure can skip re-running (and re-paying for) already-submitted cases. The Python
SDK’s
execute() derives its keys deterministically as {runId}:{caseId}:run-{runNumber} and
uses exactly this endpoint to resume.
The engine cannot know your full case list, so it reports what was submitted, not what is
missing: the legacy
missing field is always an empty array. Compute missing cases
client-side as your own expected keys minus submittedKeys.Authentication
string
required
Project API key.
Path Parameters
string
required
Run ID.
Response
string
Echoed back.
array
The
idempotencyKey of every result this run has accepted.number
submittedKeys.length.array
Always
[]. Legacy field kept for older SDK versions; the client computes missing cases
from submittedKeys.Errors
| Status | Body | Meaning |
|---|---|---|
404 | { "error": "Run not found" } | No run with this id in the project |
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
{
"error": "Run not found"
}