curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "Click Forgot Password on the login screen."
}
},
{
"main_question": {
"query": "What payment methods do you accept?"
}
}
],
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true }
}'
dataset = (
client.evaluations.datasets.builder(
"Customer Support Q3 2026",
description="Core support questions",
number_of_requests=3,
acceptance_criteria="Accurate, empathetic, resolves the issue",
rejection_criteria="Hallucinates, ignores the question",
vector_similarity=True,
jaccard_similarity=True,
)
.add_case("How do I reset my password?",
expected_results="Click Forgot Password on the login screen.")
.add_case("What payment methods do you accept?")
.publish()
)
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true },
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } },
{ "main_question": { "query": "What payment methods do you accept?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
{
"error": "name is required"
}
Custom Evaluations
Create Dataset
Create an evaluation dataset with questions and scoring configuration
POST
/
api
/
v1
/
custom-agent-evaluations
/
datasets
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "Click Forgot Password on the login screen."
}
},
{
"main_question": {
"query": "What payment methods do you accept?"
}
}
],
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true }
}'
dataset = (
client.evaluations.datasets.builder(
"Customer Support Q3 2026",
description="Core support questions",
number_of_requests=3,
acceptance_criteria="Accurate, empathetic, resolves the issue",
rejection_criteria="Hallucinates, ignores the question",
vector_similarity=True,
jaccard_similarity=True,
)
.add_case("How do I reset my password?",
expected_results="Click Forgot Password on the login screen.")
.add_case("What payment methods do you accept?")
.publish()
)
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true },
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } },
{ "main_question": { "query": "What payment methods do you accept?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
{
"error": "name is required"
}
Creates a dataset: the questions your agent will be evaluated against plus the grading
configuration (judge criteria, similarity metrics, code scorers) used when a run doesn’t bring
its own evaluation-settings config. Creation
also seeds version
v0 of the dataset’s version history.
Authentication
string
required
Project API key.
Body
string
required
Dataset display name.
array
default:"[]"
Question objects. Not validated at creation time (an empty or malformed array is accepted),
but a run against a dataset with no usable questions has nothing to score. Each question
wraps its fields in
main_question:[
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "The user should click Forgot Password on the login screen."
}
}
]
Show main_question fields
Show main_question fields
| Field | Type | Description |
|---|---|---|
query | string | The question posed to the agent |
expectedResults | string | Reference answer. Enables similarity scoring and is shown to the judge; required per-case by requiresExpected grading configs |
judgeGuideline | string | Case-specific guidance added to the judge prompt |
retrievalContext | string | Statically pinned context for {context}-referencing judge prompts (used only when the result and its linked trace carry none) |
expectedRetrievalContext | string | string[] | Reference retrieval: what a correct retriever should have fetched. Compared to the actually-retrieved context by token Jaccard similarity - no judge call |
expectedTrajectory | object | { "tools": string[], "mode": "strict" | "unordered" | "subset" | "superset" } - expected tool calls, matched against the linked trace’s actual tool sequence |
smokeTest | object | { "enabled": boolean, "count": number, "guidance": string } - generate paraphrase variants of this question at run creation for consistency testing |
splits | string[] | Named subset tags; a run created with split covers only cases tagged with it |
string
Human-readable description.
number
default:"1"
How many times each question is run per evaluation (for consistency testing).
string
What a good response looks like. Included in the LLM scoring prompt.
string
What a bad response looks like. Included in the LLM scoring prompt.
string
Scoring rubric. Included in the LLM scoring prompt.
object
Enable cosine similarity scoring against
expectedResults (requires an embeddings-capable
provider key). Only { "enabled": true } activates it; model optionally selects the
embedding model.{ "enabled": true, "model": "text-embedding-3-small" }
object
Enable Jaccard token-overlap scoring against
expectedResults: { "enabled": true }.object
Enable BLEU scoring against
expectedResults: { "enabled": true }.object
Enable ROUGE scoring against
expectedResults: { "enabled": true }.array
Deterministic code scorers run on every result. Each:
{ "id"?: string, "name": string, "code": string, "enabled"?: boolean }. Entries with empty
code are dropped; enabled defaults to true.Response
Returns201 Created with the full dataset document, keyed by _id. status is always
"published" on self-host. The enabled similarity metrics come back as top-level keys
(vectorSimilarity, jaccardSimilarity, …), matching what was sent.
Errors
| Status | Body | Meaning |
|---|---|---|
400 | { "error": "name is required" } | name missing, not a string, or blank |
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "Click Forgot Password on the login screen."
}
},
{
"main_question": {
"query": "What payment methods do you accept?"
}
}
],
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true }
}'
dataset = (
client.evaluations.datasets.builder(
"Customer Support Q3 2026",
description="Core support questions",
number_of_requests=3,
acceptance_criteria="Accurate, empathetic, resolves the issue",
rejection_criteria="Hallucinates, ignores the question",
vector_similarity=True,
jaccard_similarity=True,
)
.add_case("How do I reset my password?",
expected_results="Click Forgot Password on the login screen.")
.add_case("What payment methods do you accept?")
.publish()
)
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true },
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } },
{ "main_question": { "query": "What payment methods do you accept?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
{
"error": "name is required"
}