curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
curl -s "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: $AGENTX_API_KEY" \
| jq -r '.questions[].main_question.query'
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"numberOfRequests": 1,
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen.", "splits": ["smoke"] } },
{ "main_question": { "query": "What payment methods do you accept?" } },
{ "main_question": { "query": "How long does shipping take?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
CI Gate
Get Test Cases
Read the dataset’s questions to drive a CI evaluation
GET
/
api
/
v1
/
custom-agent-evaluations
/
datasets
/
{datasetId}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
curl -s "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: $AGENTX_API_KEY" \
| jq -r '.questions[].main_question.query'
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"numberOfRequests": 1,
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen.", "splits": ["smoke"] } },
{ "main_question": { "query": "What payment methods do you accept?" } },
{ "main_question": { "query": "How long does shipping take?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
The test cases a CI run executes are the dataset’s own
questions array - there is no separate
test-case endpoint. Fetch the dataset, iterate questions, run your agent on each
main_question.query, and submit each answer with the case’s array position as
questionIndex.
This is the same endpoint as Get Dataset; this page
covers the CI angle.
Authentication
string
required
Project API key.
Path Parameters
string
required
Dataset
_id.Using the response in CI
For each element ofquestions (0-based index i):
main_question.queryis the input to pose to your agent.- Submit the answer with
questionIndex: i(andrunNumber1 throughnumberOfRequestswhen repeating cases for consistency). main_question.splits, when present, tags the case’s named subsets - a run created with"split": "smoke"should execute only cases tagged"smoke", keeping their original indexes.expectedResults,expectedTrajectory, and the other grading fields are read server-side by the judge and scorers; your harness doesn’t need them (thoughexpectedResultsis useful for local debugging).
Errors
| Status | Body | Meaning |
|---|---|---|
404 | { "error": "Dataset not found" } | No dataset with this id in the project |
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
curl -s "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: $AGENTX_API_KEY" \
| jq -r '.questions[].main_question.query'
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"numberOfRequests": 1,
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen.", "splits": ["smoke"] } },
{ "main_question": { "query": "What payment methods do you accept?" } },
{ "main_question": { "query": "How long does shipping take?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}