curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"framework": "langchain",
"version": "v2-retrieval"
},
"runSource": "sdk"
}'
from agentx.evaluations.models import EvaluationSubject
# client.evaluations.run(dataset_id, subject, ...) drives the whole lifecycle;
# init_run alone creates the run to drive manually.
run = client.evaluations.init_run(
dataset_id="dS4tG7hNb2VxZ8kQ5wMyA",
subject=EvaluationSubject(kind="custom_agent", display_name="Customer Support Bot"),
)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
{
"error": "Dataset not found"
}
Custom Evaluations
Create Run
Start a new evaluation run against a dataset
POST
/
api
/
v1
/
custom-agent-evaluations
/
runs
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"framework": "langchain",
"version": "v2-retrieval"
},
"runSource": "sdk"
}'
from agentx.evaluations.models import EvaluationSubject
# client.evaluations.run(dataset_id, subject, ...) drives the whole lifecycle;
# init_run alone creates the run to drive manually.
run = client.evaluations.init_run(
dataset_id="dS4tG7hNb2VxZ8kQ5wMyA",
subject=EvaluationSubject(kind="custom_agent", display_name="Customer Support Bot"),
)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
{
"error": "Dataset not found"
}
Initialises a new evaluation run with status
"in_progress". Submit your agent’s outputs to it
with Submit Results, then
finalize. If any dataset case has smoke testing
enabled, the paraphrase variants are generated once here and frozen for the run’s lifetime.
Authentication
string
required
Project API key.
Body
string
required
Dataset ID to evaluate against.
string
Grade with a standalone, reusable grading config (criteria, judge prompt/model, similarity
metrics, code scorers) instead of the dataset’s own configuration. Lets one config run
against any dataset. Omit to use the dataset’s own criteria.
string
Grade with a scorer group - the group’s weighted aggregate fills
the run’s rating, judge members land as per-row verdicts, and deterministic members as scorer
rows. A group is a complete grading recipe: when it resolves,
evaluationSettingsId and
additionalScorerIds are ignored.string[]
Extra LLM judge scorer ids (up to 4, self-host) that also grade every result - one agent
execution, N verdicts. The primary scorer keeps the
rating column; each extra verdict lands
in the result row’s judgeScorerResults and rolls up into the run’s scorerBreakdown
(Get Run). Duplicates of the primary are dropped, and
an id that no longer resolves to a scorer is silently skipped (that scorer simply does not
grade the run) - unlike scorerGroupId, where an unknown id is a hard 404.object
Free-form metadata about the agent being evaluated (e.g.
kind, displayName, framework,
runtime, agentInstructions). Stored on the run and echoed back by
Get Run. A version field (or
metadata.version) tags the run for version-to-version comparison in the dashboard.string
default:"\"sdk\""
How the run was triggered, e.g.
"sdk". Trace evaluations use "trace-eval" internally and
are excluded from CI-gate baselines.object
SDK name/version info, stored for diagnostics.
string
Named case subset: run only the dataset cases tagged with this split
(
main_question.splits). The caller does the filtering when executing; the split name is
recorded on the run’s subject so split runs are visibly split runs. Original question
indexes are preserved, so per-case comparisons line up with full runs.Response
Returns201 Created.
string
Use in all subsequent calls for this run.
string
Echoed back.
string
Always
"in_progress" on creation.array | null
One group per dataset case with
smokeTest.enabled:
[{ "questionIndex": number, "variants": string[] }]. null when no case requests smoke
testing. Run each variant like a normal case and submit its result with
isSmokeTestVariant: true.Errors
| Status | Body | Meaning |
|---|---|---|
400 | { "error": "datasetId is required" } | Missing or non-string datasetId |
400 | { "error": "evaluationSettingsId must be a string" } | evaluationSettingsId present but not a string (or null) |
400 | { "error": "scorerGroupId must be a string" } | scorerGroupId present but not a string (or null) - a 400, not a silent drop, so a malformed group id can’t downgrade the run to the dataset’s own grading unnoticed |
404 | { "error": "Scorer group not found" } | No scorer group with this id in the project |
404 | { "error": "Dataset not found" } | No dataset with this id in the project |
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"framework": "langchain",
"version": "v2-retrieval"
},
"runSource": "sdk"
}'
from agentx.evaluations.models import EvaluationSubject
# client.evaluations.run(dataset_id, subject, ...) drives the whole lifecycle;
# init_run alone creates the run to drive manually.
run = client.evaluations.init_run(
dataset_id="dS4tG7hNb2VxZ8kQ5wMyA",
subject=EvaluationSubject(kind="custom_agent", display_name="Customer Support Bot"),
)
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
{
"error": "Dataset not found"
}