curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
datasets = client.evaluations.datasets.list()
for d in datasets:
print(d.id, d.name)
{
"datasets": [
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions for tier-1 agents",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question, rude",
"jaccardSimilarity": { "enabled": true },
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }
],
"status": "published",
"createdAt": "2026-08-01T09:00:00.000Z"
}
]
}
Custom Evaluations
List Datasets
List the project’s evaluation datasets
GET
/
api
/
v1
/
custom-agent-evaluations
/
datasets
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
datasets = client.evaluations.datasets.list()
for d in datasets:
print(d.id, d.name)
{
"datasets": [
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions for tier-1 agents",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question, rude",
"jaccardSimilarity": { "enabled": true },
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }
],
"status": "published",
"createdAt": "2026-08-01T09:00:00.000Z"
}
]
}
Returns every dataset in the project selected by your API key, newest first. Use the
_id of
the dataset you want as datasetId when creating a run.
Authentication
string
required
Project API key.
Response
array
Full dataset documents (including
questions), newest first. Each has:| Field | Type | Description |
|---|---|---|
_id | string | Dataset ID; pass as datasetId when creating runs |
name | string | Display name |
description | string | Description (omitted when unset) |
numberOfRequests | number | Repetitions per question (default 1) |
acceptanceCriteria | string | What a good response looks like (omitted when unset) |
rejectionCriteria | string | What a bad response looks like (omitted when unset) |
evaluationCriteria | string | Scoring rubric (omitted when unset) |
vectorSimilarity / jaccardSimilarity / bleuScore / rougeScore | object | Present only when the metric is enabled, e.g. { "enabled": true } (vectorSimilarity may also carry a model, as on Create Dataset) |
codeScorers | array | Configured code scorers (omitted when none) |
questions | array | The dataset’s question objects |
status | string | Always "published" on self-host |
createdAt | string | ISO 8601 timestamp |
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
datasets = client.evaluations.datasets.list()
for d in datasets:
print(d.id, d.name)
{
"datasets": [
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions for tier-1 agents",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question, rude",
"jaccardSimilarity": { "enabled": true },
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }
],
"status": "published",
"createdAt": "2026-08-01T09:00:00.000Z"
}
]
}