curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \ -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \ -H "Content-Type: application/json" \ -d '{ "name": "Customer Support Q3 2026", "description": "Core support questions", "numberOfRequests": 3, "acceptanceCriteria": "Accurate, empathetic, resolves the issue", "rejectionCriteria": "Hallucinates, ignores the question", "questions": [ { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }, { "main_question": { "query": "What payment methods do you accept?" } } ], "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true } }'
dataset = ( client.evaluations.datasets.builder( "Customer Support Q3 2026", description="Core support questions", number_of_requests=3, acceptance_criteria="Accurate, empathetic, resolves the issue", rejection_criteria="Hallucinates, ignores the question", vector_similarity=True, jaccard_similarity=True, ) .add_case("How do I reset my password?", expected_results="Click Forgot Password on the login screen.") .add_case("What payment methods do you accept?") .publish())
{ "_id": "dS4tG7hNb2VxZ8kQ5wMyA", "name": "Customer Support Q3 2026", "description": "Core support questions", "numberOfRequests": 3, "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true }, "acceptanceCriteria": "Accurate, empathetic, resolves the issue", "rejectionCriteria": "Hallucinates, ignores the question", "questions": [ { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }, { "main_question": { "query": "What payment methods do you accept?" } } ], "status": "published", "createdAt": "2026-08-27T10:00:00.000Z"}
{ "error": "name is required"}
Custom Evaluations
Create Dataset
Create an evaluation dataset with questions and scoring configuration
POST
/
api
/
v1
/
custom-agent-evaluations
/
datasets
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \ -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \ -H "Content-Type: application/json" \ -d '{ "name": "Customer Support Q3 2026", "description": "Core support questions", "numberOfRequests": 3, "acceptanceCriteria": "Accurate, empathetic, resolves the issue", "rejectionCriteria": "Hallucinates, ignores the question", "questions": [ { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }, { "main_question": { "query": "What payment methods do you accept?" } } ], "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true } }'
dataset = ( client.evaluations.datasets.builder( "Customer Support Q3 2026", description="Core support questions", number_of_requests=3, acceptance_criteria="Accurate, empathetic, resolves the issue", rejection_criteria="Hallucinates, ignores the question", vector_similarity=True, jaccard_similarity=True, ) .add_case("How do I reset my password?", expected_results="Click Forgot Password on the login screen.") .add_case("What payment methods do you accept?") .publish())
{ "_id": "dS4tG7hNb2VxZ8kQ5wMyA", "name": "Customer Support Q3 2026", "description": "Core support questions", "numberOfRequests": 3, "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true }, "acceptanceCriteria": "Accurate, empathetic, resolves the issue", "rejectionCriteria": "Hallucinates, ignores the question", "questions": [ { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }, { "main_question": { "query": "What payment methods do you accept?" } } ], "status": "published", "createdAt": "2026-08-27T10:00:00.000Z"}
{ "error": "name is required"}
Creates a dataset: the questions your agent will be evaluated against plus the grading
configuration (judge criteria, similarity metrics, code scorers) used when a run doesn’t bring
its own evaluation-settings config. Creation
also seeds version v0 of the dataset’s version history.
Authentication
string
required
Project API key.
Body
string
required
Dataset display name.
array
default:"[]"
Question objects. Not validated at creation time (an empty or malformed array is accepted),
but a run against a dataset with no usable questions has nothing to score. Each question
wraps its fields in main_question:
[ { "main_question": { "query": "How do I reset my password?", "expectedResults": "The user should click Forgot Password on the login screen." } }]
Show main_question fields
Field
Type
Description
query
string
The question posed to the agent
expectedResults
string
Reference answer. Enables similarity scoring and is shown to the judge; required per-case by requiresExpected grading configs
judgeGuideline
string
Case-specific guidance added to the judge prompt
retrievalContext
string
Statically pinned context for {context}-referencing judge prompts (used only when the result and its linked trace carry none)
expectedRetrievalContext
string | string[]
Reference retrieval: what a correct retriever should have fetched. Compared to the actually-retrieved context by token Jaccard similarity - no judge call
expectedTrajectory
object
{ "tools": string[], "mode": "strict" | "unordered" | "subset" | "superset" } - expected tool calls, matched against the linked trace’s actual tool sequence
smokeTest
object
{ "enabled": boolean, "count": number, "guidance": string } - generate paraphrase variants of this question at run creation for consistency testing
splits
string[]
Named subset tags; a run created with split covers only cases tagged with it
string
Human-readable description.
number
default:"1"
How many times each question is run per evaluation (for consistency testing).
string
What a good response looks like. Included in the LLM scoring prompt.
string
What a bad response looks like. Included in the LLM scoring prompt.
string
Scoring rubric. Included in the LLM scoring prompt.
object
Enable cosine similarity scoring against expectedResults (requires an embeddings-capable
provider key). Only { "enabled": true } activates it; model optionally selects the
embedding model.
Enable Jaccard token-overlap scoring against expectedResults: { "enabled": true }.
object
Enable BLEU scoring against expectedResults: { "enabled": true }.
object
Enable ROUGE scoring against expectedResults: { "enabled": true }.
array
Deterministic code scorers run on every result. Each:
{ "id"?: string, "name": string, "code": string, "enabled"?: boolean }. Entries with empty
code are dropped; enabled defaults to true.
Response
Returns 201 Created with the full dataset document, keyed by _id. status is always
"published" on self-host. The enabled similarity metrics come back as top-level keys
(vectorSimilarity, jaccardSimilarity, …), matching what was sent.
Errors
Status
Body
Meaning
400
{ "error": "name is required" }
name missing, not a string, or blank
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \ -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \ -H "Content-Type: application/json" \ -d '{ "name": "Customer Support Q3 2026", "description": "Core support questions", "numberOfRequests": 3, "acceptanceCriteria": "Accurate, empathetic, resolves the issue", "rejectionCriteria": "Hallucinates, ignores the question", "questions": [ { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }, { "main_question": { "query": "What payment methods do you accept?" } } ], "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true } }'
dataset = ( client.evaluations.datasets.builder( "Customer Support Q3 2026", description="Core support questions", number_of_requests=3, acceptance_criteria="Accurate, empathetic, resolves the issue", rejection_criteria="Hallucinates, ignores the question", vector_similarity=True, jaccard_similarity=True, ) .add_case("How do I reset my password?", expected_results="Click Forgot Password on the login screen.") .add_case("What payment methods do you accept?") .publish())
{ "_id": "dS4tG7hNb2VxZ8kQ5wMyA", "name": "Customer Support Q3 2026", "description": "Core support questions", "numberOfRequests": 3, "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true }, "acceptanceCriteria": "Accurate, empathetic, resolves the issue", "rejectionCriteria": "Hallucinates, ignores the question", "questions": [ { "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }, { "main_question": { "query": "What payment methods do you accept?" } } ], "status": "published", "createdAt": "2026-08-27T10:00:00.000Z"}
{ "error": "name is required"}
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"AgentX","colors":{"primary":"#2A5699","light":"#5C8DCC","dark":"#1A3F73"},"logo":{"light":"https://mintcdn.com/agentx-ffe8d995/ZTzZ3HDQJ-RJxymw/logo/light.png?fit=max\u0026auto=format\u0026n=ZTzZ3HDQJ-RJxymw\u0026q=85\u0026s=c1e30c949dc1cc2aaa3a137c1ee5c77f","dark":"https://mintcdn.com/agentx-ffe8d995/ZTzZ3HDQJ-RJxymw/logo/dark.png?fit=max\u0026auto=format\u0026n=ZTzZ3HDQJ-RJxymw\u0026q=85\u0026s=35a521e090f7875c15bef7ae392af788"},"favicon":"https://mintcdn.com/agentx-ffe8d995/FRrvmlkH0m2BUb9c/logo/favicon.png?fit=max\u0026auto=format\u0026n=FRrvmlkH0m2BUb9c\u0026q=85\u0026s=e87d636d38c8e6e187b4be187e72381d","api":{"mdx":{"auth":{"method":"key","name":"x-api-key"},"server":"http://localhost:4700"}},"navbar":{"primary":{"type":"github","href":"https://github.com/AgentX-ai/AgentX-trace-eval"}},"footer":{"socials":{"website":"https://www.agentx.so/mcp/ai-evaluation","github":"https://github.com/AgentX-ai/AgentX-trace-eval"}},"styling":{"latex":true},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"agentx-ffe8d995","hasAudioNarration":false,"actualSubdomain":"agentx-ffe8d995","customDomains":["developers.agentx.so"],"gitSource":{"type":"github","owner":"agentx-ai","repo":"agentx-mintlify","deployBranch":"main","contentDirectory":"","isPrivate":true,"mountPath":"","label":""},"entitlements":{"AI_CHAT":{"status":"DISABLED"}},"buildId":"6ac683381cb9dd011c327d04","clientVersion":"0.0.4270","searchFilterCounts":[{"language":"default","tag":"API Reference","count":25},{"language":"default","tag":"Offline Evaluation","count":24},{"language":"default","tag":"Trace","count":17},{"language":"default","tag":"Scorers \u0026 Monitoring","count":16},{"language":"default","tag":"Custom Evaluations","count":13},{"language":"default","tag":"Documentation","count":13},{"language":"default","tag":"Framework Integrations","count":12},{"language":"default","tag":"Framework Examples","count":10},{"language":"default","tag":"Tracing","count":8},{"language":"default","tag":"Live Traffic Scoring","count":7}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableContextualStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"site-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[]}
{"page":{"pagePath":"api-reference/custom-eval/create-dataset","slug":"/api-reference/custom-eval/create-dataset","pageMetadata":{"title":"Create Dataset","api":"POST /api/v1/custom-agent-evaluations/datasets","description":"Create an evaluation dataset with questions and scoring configuration","href":"/api-reference/custom-eval/create-dataset"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Create an evaluation dataset with questions and scoring configuration\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,p:`p`,pre:`pre`,span:`span`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,..._provideComponents(),...props.components},{CodeBlock,Expandable,Heading,ParamField,Table}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Expandable)_missingMdxReference(`Expandable`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!Table)_missingMdxReference(`Table`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Creates a dataset: the questions your agent will be evaluated against plus the grading\nconfiguration (judge criteria, similarity metrics, code scorers) used when a run doesn’t bring\nits own `,_jsx(_components.a,{href:`/api-reference/custom-eval/overview#endpoints`,children:`evaluation-settings config`}),`. Creation\nalso seeds version `,_jsx(_components.code,{children:`v0`}),` of the dataset’s version history.`]}),`\n`,_jsx(Heading,{level:`2`,id:`authentication`,children:`Authentication`}),`\n`,_jsx(ParamField,{header:`x-api-key`,type:`string`,required:true,id:`param-x-api-key`,children:_jsx(_components.p,{children:`Project API key.`})}),`\n`,_jsx(Heading,{level:`2`,id:`body`,children:`Body`}),`\n`,_jsx(ParamField,{body:`name`,type:`string`,required:true,id:`param-name`,children:_jsx(_components.p,{children:`Dataset display name.`})}),`\n`,_jsxs(ParamField,{body:`questions`,type:`array`,default:`[]`,id:`param-questions`,children:[_jsxs(_components.p,{children:[`Question objects. Not validated at creation time (an empty or malformed array is accepted),\nbut a run against a dataset with no usable questions has nothing to score. Each question\nwraps its fields in `,_jsx(_components.code,{children:`main_question`}),`:`]}),_jsx(CodeBlock,{filename:``,numberOfLines:`8`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`8`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`[`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"main_question\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"query\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"How do I reset my password?\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"expectedResults\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"The user should click Forgot Password on the login screen.\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})}),`\n`]})})}),_jsx(Expandable,{title:`main_question fields`,children:_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Field`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`query`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`The question posed to the agent`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expectedResults`})}),_jsx(_components.td,{children:`string`}),_jsxs(_components.td,{children:[`Reference answer. Enables similarity scoring and is shown to the judge; required per-case by `,_jsx(_components.code,{children:`requiresExpected`}),` grading configs`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`judgeGuideline`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`Case-specific guidance added to the judge prompt`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`retrievalContext`})}),_jsx(_components.td,{children:`string`}),_jsxs(_components.td,{children:[`Statically pinned context for `,_jsx(_components.code,{children:`{context}`}),`-referencing judge prompts (used only when the result and its linked trace carry none)`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expectedRetrievalContext`})}),_jsx(_components.td,{children:`string | string[]`}),_jsx(_components.td,{children:`Reference retrieval: what a correct retriever should have fetched. Compared to the actually-retrieved context by token Jaccard similarity - no judge call`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expectedTrajectory`})}),_jsx(_components.td,{children:`object`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`{ \"tools\": string[], \"mode\": \"strict\" | \"unordered\" | \"subset\" | \"superset\" }`}),` - expected tool calls, matched against the linked trace’s actual tool sequence`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`smokeTest`})}),_jsx(_components.td,{children:`object`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`{ \"enabled\": boolean, \"count\": number, \"guidance\": string }`}),` - generate paraphrase variants of this question at run creation for consistency testing`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`splits`})}),_jsx(_components.td,{children:`string[]`}),_jsxs(_components.td,{children:[`Named subset tags; a run created with `,_jsx(_components.code,{children:`split`}),` covers only cases tagged with it`]})]})]})]})})]}),`\n`,_jsx(ParamField,{body:`description`,type:`string`,id:`param-description`,children:_jsx(_components.p,{children:`Human-readable description.`})}),`\n`,_jsx(ParamField,{body:`numberOfRequests`,type:`number`,default:`1`,id:`param-number-of-requests`,children:_jsx(_components.p,{children:`How many times each question is run per evaluation (for consistency testing).`})}),`\n`,_jsx(ParamField,{body:`acceptanceCriteria`,type:`string`,id:`param-acceptance-criteria`,children:_jsx(_components.p,{children:`What a good response looks like. Included in the LLM scoring prompt.`})}),`\n`,_jsx(ParamField,{body:`rejectionCriteria`,type:`string`,id:`param-rejection-criteria`,children:_jsx(_components.p,{children:`What a bad response looks like. Included in the LLM scoring prompt.`})}),`\n`,_jsx(ParamField,{body:`evaluationCriteria`,type:`string`,id:`param-evaluation-criteria`,children:_jsx(_components.p,{children:`Scoring rubric. Included in the LLM scoring prompt.`})}),`\n`,_jsxs(ParamField,{body:`vectorSimilarity`,type:`object`,id:`param-vector-similarity`,children:[_jsxs(_components.p,{children:[`Enable cosine similarity scoring against `,_jsx(_components.code,{children:`expectedResults`}),` (requires an embeddings-capable\nprovider key). Only `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),` activates it; `,_jsx(_components.code,{children:`model`}),` optionally selects the\nembedding model.`]}),_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{ `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"enabled\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`true`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"model\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"text-embedding-3-small\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})]}),`\n`]})})})]}),`\n`,_jsx(ParamField,{body:`jaccardSimilarity`,type:`object`,id:`param-jaccard-similarity`,children:_jsxs(_components.p,{children:[`Enable Jaccard token-overlap scoring against `,_jsx(_components.code,{children:`expectedResults`}),`: `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),`.`]})}),`\n`,_jsx(ParamField,{body:`bleuScore`,type:`object`,id:`param-bleu-score`,children:_jsxs(_components.p,{children:[`Enable BLEU scoring against `,_jsx(_components.code,{children:`expectedResults`}),`: `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),`.`]})}),`\n`,_jsx(ParamField,{body:`rougeScore`,type:`object`,id:`param-rouge-score`,children:_jsxs(_components.p,{children:[`Enable ROUGE scoring against `,_jsx(_components.code,{children:`expectedResults`}),`: `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),`.`]})}),`\n`,_jsx(ParamField,{body:`codeScorers`,type:`array`,id:`param-code-scorers`,children:_jsxs(_components.p,{children:[`Deterministic code scorers run on every result. Each:\n`,_jsx(_components.code,{children:`{ \"id\"?: string, \"name\": string, \"code\": string, \"enabled\"?: boolean }`}),`. Entries with empty\n`,_jsx(_components.code,{children:`code`}),` are dropped; `,_jsx(_components.code,{children:`enabled`}),` defaults to `,_jsx(_components.code,{children:`true`}),`.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`response`,children:`Response`}),`\n`,_jsxs(_components.p,{children:[`Returns `,_jsx(_components.code,{children:`201 Created`}),` with the full dataset document, keyed by `,_jsx(_components.code,{children:`_id`}),`. `,_jsx(_components.code,{children:`status`}),` is always\n`,_jsx(_components.code,{children:`\"published\"`}),` on self-host. The enabled similarity metrics come back as top-level keys\n(`,_jsx(_components.code,{children:`vectorSimilarity`}),`, `,_jsx(_components.code,{children:`jaccardSimilarity`}),`, …), matching what was sent.`]}),`\n`,_jsx(Heading,{level:`2`,id:`errors`,children:`Errors`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Status`}),_jsx(_components.th,{children:`Body`}),_jsx(_components.th,{children:`Meaning`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{\"data-numeric\":`true`,children:_jsx(_components.code,{children:`400`})}),_jsx(_components.td,{children:_jsx(_components.code,{children:`{ \"error\": \"name is required\" }`})}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`name`}),` missing, not a string, or blank`]})]})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Create Dataset","api":"POST /api/v1/custom-agent-evaluations/datasets","description":"Create an evaluation dataset with questions and scoring configuration","href":"/api-reference/custom-eval/create-dataset"}}},"mdxSourceWithNoJs":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,p:`p`,pre:`pre`,span:`span`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,..._provideComponents(),...props.components},{CodeBlock,Expandable,Heading,ParamField,RequestExample,ResponseExample,Table}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Expandable)_missingMdxReference(`Expandable`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!RequestExample)_missingMdxReference(`RequestExample`,true);if(!ResponseExample)_missingMdxReference(`ResponseExample`,true);if(!Table)_missingMdxReference(`Table`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Creates a dataset: the questions your agent will be evaluated against plus the grading\nconfiguration (judge criteria, similarity metrics, code scorers) used when a run doesn’t bring\nits own `,_jsx(_components.a,{href:`/api-reference/custom-eval/overview#endpoints`,children:`evaluation-settings config`}),`. Creation\nalso seeds version `,_jsx(_components.code,{children:`v0`}),` of the dataset’s version history.`]}),`\n`,_jsx(Heading,{level:`2`,id:`authentication`,children:`Authentication`}),`\n`,_jsx(ParamField,{header:`x-api-key`,type:`string`,required:true,id:`param-x-api-key`,children:_jsx(_components.p,{children:`Project API key.`})}),`\n`,_jsx(Heading,{level:`2`,id:`body`,children:`Body`}),`\n`,_jsx(ParamField,{body:`name`,type:`string`,required:true,id:`param-name`,children:_jsx(_components.p,{children:`Dataset display name.`})}),`\n`,_jsxs(ParamField,{body:`questions`,type:`array`,default:`[]`,id:`param-questions`,children:[_jsxs(_components.p,{children:[`Question objects. Not validated at creation time (an empty or malformed array is accepted),\nbut a run against a dataset with no usable questions has nothing to score. Each question\nwraps its fields in `,_jsx(_components.code,{children:`main_question`}),`:`]}),_jsx(CodeBlock,{filename:``,numberOfLines:`8`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`8`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`[`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"main_question\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"query\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"How do I reset my password?\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"expectedResults\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"The user should click Forgot Password on the login screen.\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})}),`\n`]})})}),_jsx(Expandable,{title:`main_question fields`,children:_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Field`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`query`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`The question posed to the agent`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expectedResults`})}),_jsx(_components.td,{children:`string`}),_jsxs(_components.td,{children:[`Reference answer. Enables similarity scoring and is shown to the judge; required per-case by `,_jsx(_components.code,{children:`requiresExpected`}),` grading configs`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`judgeGuideline`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`Case-specific guidance added to the judge prompt`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`retrievalContext`})}),_jsx(_components.td,{children:`string`}),_jsxs(_components.td,{children:[`Statically pinned context for `,_jsx(_components.code,{children:`{context}`}),`-referencing judge prompts (used only when the result and its linked trace carry none)`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expectedRetrievalContext`})}),_jsx(_components.td,{children:`string | string[]`}),_jsx(_components.td,{children:`Reference retrieval: what a correct retriever should have fetched. Compared to the actually-retrieved context by token Jaccard similarity - no judge call`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expectedTrajectory`})}),_jsx(_components.td,{children:`object`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`{ \"tools\": string[], \"mode\": \"strict\" | \"unordered\" | \"subset\" | \"superset\" }`}),` - expected tool calls, matched against the linked trace’s actual tool sequence`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`smokeTest`})}),_jsx(_components.td,{children:`object`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`{ \"enabled\": boolean, \"count\": number, \"guidance\": string }`}),` - generate paraphrase variants of this question at run creation for consistency testing`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`splits`})}),_jsx(_components.td,{children:`string[]`}),_jsxs(_components.td,{children:[`Named subset tags; a run created with `,_jsx(_components.code,{children:`split`}),` covers only cases tagged with it`]})]})]})]})})]}),`\n`,_jsx(ParamField,{body:`description`,type:`string`,id:`param-description`,children:_jsx(_components.p,{children:`Human-readable description.`})}),`\n`,_jsx(ParamField,{body:`numberOfRequests`,type:`number`,default:`1`,id:`param-number-of-requests`,children:_jsx(_components.p,{children:`How many times each question is run per evaluation (for consistency testing).`})}),`\n`,_jsx(ParamField,{body:`acceptanceCriteria`,type:`string`,id:`param-acceptance-criteria`,children:_jsx(_components.p,{children:`What a good response looks like. Included in the LLM scoring prompt.`})}),`\n`,_jsx(ParamField,{body:`rejectionCriteria`,type:`string`,id:`param-rejection-criteria`,children:_jsx(_components.p,{children:`What a bad response looks like. Included in the LLM scoring prompt.`})}),`\n`,_jsx(ParamField,{body:`evaluationCriteria`,type:`string`,id:`param-evaluation-criteria`,children:_jsx(_components.p,{children:`Scoring rubric. Included in the LLM scoring prompt.`})}),`\n`,_jsxs(ParamField,{body:`vectorSimilarity`,type:`object`,id:`param-vector-similarity`,children:[_jsxs(_components.p,{children:[`Enable cosine similarity scoring against `,_jsx(_components.code,{children:`expectedResults`}),` (requires an embeddings-capable\nprovider key). Only `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),` activates it; `,_jsx(_components.code,{children:`model`}),` optionally selects the\nembedding model.`]}),_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{ `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"enabled\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`true`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"model\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"text-embedding-3-small\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})]}),`\n`]})})})]}),`\n`,_jsx(ParamField,{body:`jaccardSimilarity`,type:`object`,id:`param-jaccard-similarity`,children:_jsxs(_components.p,{children:[`Enable Jaccard token-overlap scoring against `,_jsx(_components.code,{children:`expectedResults`}),`: `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),`.`]})}),`\n`,_jsx(ParamField,{body:`bleuScore`,type:`object`,id:`param-bleu-score`,children:_jsxs(_components.p,{children:[`Enable BLEU scoring against `,_jsx(_components.code,{children:`expectedResults`}),`: `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),`.`]})}),`\n`,_jsx(ParamField,{body:`rougeScore`,type:`object`,id:`param-rouge-score`,children:_jsxs(_components.p,{children:[`Enable ROUGE scoring against `,_jsx(_components.code,{children:`expectedResults`}),`: `,_jsx(_components.code,{children:`{ \"enabled\": true }`}),`.`]})}),`\n`,_jsx(ParamField,{body:`codeScorers`,type:`array`,id:`param-code-scorers`,children:_jsxs(_components.p,{children:[`Deterministic code scorers run on every result. Each:\n`,_jsx(_components.code,{children:`{ \"id\"?: string, \"name\": string, \"code\": string, \"enabled\"?: boolean }`}),`. Entries with empty\n`,_jsx(_components.code,{children:`code`}),` are dropped; `,_jsx(_components.code,{children:`enabled`}),` defaults to `,_jsx(_components.code,{children:`true`}),`.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`response`,children:`Response`}),`\n`,_jsxs(_components.p,{children:[`Returns `,_jsx(_components.code,{children:`201 Created`}),` with the full dataset document, keyed by `,_jsx(_components.code,{children:`_id`}),`. `,_jsx(_components.code,{children:`status`}),` is always\n`,_jsx(_components.code,{children:`\"published\"`}),` on self-host. The enabled similarity metrics come back as top-level keys\n(`,_jsx(_components.code,{children:`vectorSimilarity`}),`, `,_jsx(_components.code,{children:`jaccardSimilarity`}),`, …), matching what was sent.`]}),`\n`,_jsx(Heading,{level:`2`,id:`errors`,children:`Errors`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Status`}),_jsx(_components.th,{children:`Body`}),_jsx(_components.th,{children:`Meaning`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{\"data-numeric\":`true`,children:_jsx(_components.code,{children:`400`})}),_jsx(_components.td,{children:_jsx(_components.code,{children:`{ \"error\": \"name is required\" }`})}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`name`}),` missing, not a string, or blank`]})]})})]}),`\n`,_jsxs(RequestExample,{children:[_jsx(CodeBlock,{filename:`cURL`,numberOfLines:`25`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`25`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`curl`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -X`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` POST`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` http://localhost:4700/api/v1/custom-agent-evaluations/datasets`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Content-Type: application/json\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -d`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` '{`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\": \"Customer Support Q3 2026\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"description\": \"Core support questions\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"numberOfRequests\": 3,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"acceptanceCriteria\": \"Accurate, empathetic, resolves the issue\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"rejectionCriteria\": \"Hallucinates, ignores the question\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"questions\": [`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"main_question\": {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"query\": \"How do I reset my password?\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"expectedResults\": \"Click Forgot Password on the login screen.\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"main_question\": {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"query\": \"What payment methods do you accept?\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"vectorSimilarity\": { \"enabled\": true },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"jaccardSimilarity\": { \"enabled\": true }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }'`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`Python SDK`,numberOfLines:`15`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`15`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dataset `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` client.evaluations.datasets.builder(`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Customer Support Q3 2026\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` description`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Core support questions\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` number_of_requests`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` acceptance_criteria`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Accurate, empathetic, resolves the issue\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` rejection_criteria`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Hallucinates, ignores the question\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` vector_similarity`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` jaccard_similarity`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` .add_case(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"How do I reset my password?\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` expected_results`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Click Forgot Password on the login screen.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` .add_case(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"What payment methods do you accept?\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` .publish()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})}),`\n`]})})})]}),`\n`,_jsxs(ResponseExample,{children:[_jsx(CodeBlock,{filename:`201 Created`,numberOfLines:`16`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`16`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"_id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"dS4tG7hNb2VxZ8kQ5wMyA\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Customer Support Q3 2026\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"description\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Core support questions\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"numberOfRequests\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"vectorSimilarity\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: { `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"enabled\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`true`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"jaccardSimilarity\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: { `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"enabled\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`true`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"acceptanceCriteria\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Accurate, empathetic, resolves the issue\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"rejectionCriteria\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Hallucinates, ignores the question\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"questions\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` { `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"main_question\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: { `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"query\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"How do I reset my password?\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"expectedResults\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Click Forgot Password on the login screen.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` } },`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` { `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"main_question\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: { `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"query\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"What payment methods do you accept?\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` } }`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ],`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"status\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"published\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"createdAt\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"2026-08-27T10:00:00.000Z\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`400 Missing name`,numberOfLines:`3`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`3`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"error\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"name is required\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Create Dataset","api":"POST /api/v1/custom-agent-evaluations/datasets","description":"Create an evaluation dataset with questions and scoring configuration","href":"/api-reference/custom-eval/create-dataset"}}}},"mdxExtracts":{"tableOfContents":[{"title":"Authentication","slug":"authentication","depth":2,"children":[]},{"title":"Body","slug":"body","depth":2,"children":[]},{"title":"Response","slug":"response","depth":2,"children":[]},{"title":"Errors","slug":"errors","depth":2,"children":[]}],"codeExamples":{"request":{"type":"RequestExample","children":[{"filename":"cURL","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"shellscript\">\u003ccode language=\"shellscript\" numberOfLines=\"25\">\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#DCDCAA\">curl\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -X\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> POST\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> http://localhost:4700/api/v1/custom-agent-evaluations/datasets\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -H\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -H\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Content-Type: application/json\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -d\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> '{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"name\": \"Customer Support Q3 2026\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"description\": \"Core support questions\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"numberOfRequests\": 3,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"acceptanceCriteria\": \"Accurate, empathetic, resolves the issue\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"rejectionCriteria\": \"Hallucinates, ignores the question\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"questions\": [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"main_question\": {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"query\": \"How do I reset my password?\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"expectedResults\": \"Click Forgot Password on the login screen.\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"main_question\": {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"query\": \"What payment methods do you accept?\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> ],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"vectorSimilarity\": { \"enabled\": true },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"jaccardSimilarity\": { \"enabled\": true }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }'\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"Python SDK","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"python\">\u003ccode language=\"python\" numberOfLines=\"15\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">dataset \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> (\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> client.evaluations.datasets.builder(\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Customer Support Q3 2026\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> description\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Core support questions\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> number_of_requests\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\">3\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> acceptance_criteria\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Accurate, empathetic, resolves the issue\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> rejection_criteria\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Hallucinates, ignores the question\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> vector_similarity\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">True\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> jaccard_similarity\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">True\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> )\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> .add_case(\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"How do I reset my password?\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> expected_results\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Click Forgot Password on the login screen.\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">)\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> .add_case(\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"What payment methods do you accept?\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">)\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> .publish()\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">)\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"}],"attributes":[]},"response":{"type":"ResponseExample","children":[{"filename":"201 Created","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"16\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"_id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"dS4tG7hNb2VxZ8kQ5wMyA\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"name\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Customer Support Q3 2026\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"description\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Core support questions\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"numberOfRequests\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\">3\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"vectorSimilarity\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: { \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"enabled\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">true\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"jaccardSimilarity\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: { \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"enabled\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">true\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"acceptanceCriteria\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Accurate, empathetic, resolves the issue\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"rejectionCriteria\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Hallucinates, ignores the question\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"questions\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> { \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"main_question\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: { \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"query\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"How do I reset my password?\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"expectedResults\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Click Forgot Password on the login screen.\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> } },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> { \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"main_question\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: { \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"query\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"What payment methods do you accept?\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> } }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"status\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"published\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"createdAt\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"2026-08-27T10:00:00.000Z\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"400 Missing name","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"3\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"error\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"name is required\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"}],"attributes":[]}},"endpoint":{"title":"Create Dataset","path":"/api/v1/custom-agent-evaluations/datasets","servers":[{"url":"http://localhost:4700"}],"method":"post","request":{"security":[{"title":"API Key Auth","parameters":{"header":{"x-api-key":{"type":"apiKey"}},"query":{},"cookie":{}}}],"parameters":{"query":{},"header":{"x-api-key":{"schema":[{"type":"string","description":"Project API key.","required":true}]}},"cookie":{},"path":{}},"body":{"application/json":{"schemaArray":[{"type":"object","properties":{"name":[{"type":"string","description":"Dataset display name.","required":true}],"questions":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"Question objects. Not validated at creation time (an empty or malformed array is accepted), but a run against a dataset with no usable questions has nothing to score. Each question wraps its fields in `main_question` :"}],"description":[{"type":"string","description":"Human-readable description."}],"numberOfRequests":[{"type":"number","description":"How many times each question is run per evaluation (for consistency testing)."}],"acceptanceCriteria":[{"type":"string","description":"What a good response looks like. Included in the LLM scoring prompt."}],"rejectionCriteria":[{"type":"string","description":"What a bad response looks like. Included in the LLM scoring prompt."}],"evaluationCriteria":[{"type":"string","description":"Scoring rubric. Included in the LLM scoring prompt."}],"vectorSimilarity":[{"type":"object","properties":{},"description":"Enable cosine similarity scoring against `expectedResults` (requires an embeddings-capable provider key). Only `{ \"enabled\": true }` activates it; `model` optionally selects the embedding model."}],"jaccardSimilarity":[{"type":"object","properties":{},"description":"Enable Jaccard token-overlap scoring against `expectedResults` : `{ \"enabled\": true }` ."}],"bleuScore":[{"type":"object","properties":{},"description":"Enable BLEU scoring against `expectedResults` : `{ \"enabled\": true }` ."}],"rougeScore":[{"type":"object","properties":{},"description":"Enable ROUGE scoring against `expectedResults` : `{ \"enabled\": true }` ."}],"codeScorers":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"Deterministic code scorers run on every result. Each: `{ \"id\"?: string, \"name\": string, \"code\": string, \"enabled\"?: boolean }` . Entries with empty `code` are dropped; `enabled` defaults to `true` ."}]}}],"examples":{"example":{"value":{"name":"\u003cstring>","questions":[{}],"description":"\u003cstring>","numberOfRequests":123,"acceptanceCriteria":"\u003cstring>","rejectionCriteria":"\u003cstring>","evaluationCriteria":"\u003cstring>","vectorSimilarity":{},"jaccardSimilarity":{},"bleuScore":{},"rougeScore":{},"codeScorers":[{}]}}}}}},"response":{},"deprecated":false}},"jsLinks":[],"subdomain":"agentx-ffe8d995","siteName":"AgentX","title":"Create Dataset - AgentX","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-09-02T05:07:32.679Z","originalFileLocation":"api-reference/custom-eval/create-dataset.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Create an evaluation dataset with questions and scoring configuration\">\u003cmeta data-mint-head name=\"application-name\" content=\"AgentX\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Create Dataset - AgentX\">\u003cmeta data-mint-head property=\"og:description\" content=\"Create an evaluation dataset with questions and scoring configuration\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"AgentX\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://developers.agentx.so/api-reference/custom-eval/create-dataset\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://agentx-ffe8d995.mintlify.site/_mintlify/api/og/v2?division=Custom+Evaluations\u0026amp;title=Create+Dataset\u0026amp;description=Create+an+evaluation+dataset+with+questions+and+scoring+configuration\u0026amp;theme=2ebdc38e645cd0f197f78a50\u0026amp;signature=62a1cd0f1764e146330c307f6b0ab862b9d52bc4d168ad5797c47de1bc131261\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Create Dataset - AgentX\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Create an evaluation dataset with questions and scoring configuration\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://agentx-ffe8d995.mintlify.site/_mintlify/api/og/v2?division=Custom+Evaluations\u0026amp;title=Create+Dataset\u0026amp;description=Create+an+evaluation+dataset+with+questions+and+scoring+configuration\u0026amp;theme=2ebdc38e645cd0f197f78a50\u0026amp;signature=62a1cd0f1764e146330c307f6b0ab862b9d52bc4d168ad5797c47de1bc131261\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/agentx-ffe8d995/sticQuUfAA1h_xwo/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"AgentX\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#2A5699\">\u003clink data-mint-head rel=\"canonical\" href=\"https://developers.agentx.so/api-reference/custom-eval/create-dataset\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/api-reference/custom-eval/create-dataset.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://developers.agentx.so/#organization\",\"name\":\"AgentX\",\"url\":\"https://developers.agentx.so\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/agentx-ffe8d995/ZTzZ3HDQJ-RJxymw/logo/light.png?fit=max\u0026auto=format\u0026n=ZTzZ3HDQJ-RJxymw\u0026q=85\u0026s=c1e30c949dc1cc2aaa3a137c1ee5c77f\"}},{\"@type\":\"WebSite\",\"@id\":\"https://developers.agentx.so#website\",\"name\":\"AgentX\",\"url\":\"https://developers.agentx.so\",\"publisher\":{\"@id\":\"https://developers.agentx.so/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset#webpage\",\"url\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset\",\"name\":\"Create Dataset\",\"description\":\"Create an evaluation dataset with questions and scoring configuration\",\"dateModified\":\"2026-09-02T05:07:32.679Z\",\"isPartOf\":{\"@id\":\"https://developers.agentx.so#website\"},\"breadcrumb\":{\"@id\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Custom Evaluations\",\"item\":\"https://developers.agentx.so/api-reference/custom-eval/overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Create Dataset\",\"item\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset\"}]},{\"@type\":[\"TechArticle\",\"APIReference\"],\"@id\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset#apireference\",\"headline\":\"Create Dataset\",\"name\":\"Create Dataset\",\"description\":\"Create an evaluation dataset with questions and scoring configuration\",\"url\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset\",\"mainEntityOfPage\":{\"@id\":\"https://developers.agentx.so/api-reference/custom-eval/create-dataset#webpage\"},\"dateModified\":\"2026-09-02T05:07:32.679Z\",\"publisher\":{\"@id\":\"https://developers.agentx.so/#organization\"},\"isPartOf\":{\"@id\":\"https://developers.agentx.so#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"AgentX\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"AgentX\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/api-reference/custom-eval/create-dataset.md","siteName":"AgentX","nav":{"global":null,"tabs":[{"tab":"Documentation","pages":[{"title":"Documentation","href":"/introduction"}]},{"tab":"Trace","pages":[{"title":"Trace","href":"/sdk/tracing"}]},{"tab":"Scorers \u0026 Monitoring","pages":[{"title":"Scorers \u0026 Monitoring","href":"/monitor/scorers"}]},{"tab":"Offline Evaluation","pages":[{"title":"Offline Evaluation","href":"/sdk/evaluations/overview"}]},{"tab":"Improve","pages":[{"title":"Improve","href":"/improve/tool-schemas"}]},{"tab":"CI/CD","pages":[{"title":"CI/CD","href":"/integrations/self-host-ci"}]},{"tab":"API Reference","groups":[{"group":"Tracing","pages":[{"title":"Submit Trace","api":"POST /api/v1/ingest/traces","href":"/api-reference/tracing/submit-trace"},{"title":"List Traces","api":"GET /api/v1/ingest/traces","href":"/api-reference/tracing/list-traces"},{"title":"Evaluate Trace","api":"POST /api/v1/ingest/traces/{traceId}/evaluate","href":"/api-reference/tracing/evaluate-trace"}]},{"group":"Custom Evaluations","pages":[{"title":"Custom Evaluations Overview","description":"Run evaluations programmatically against any external agent and score with the LLM judge","href":"/api-reference/custom-eval/overview"},{"title":"Create Dataset","api":"POST /api/v1/custom-agent-evaluations/datasets","href":"/api-reference/custom-eval/create-dataset"},{"title":"List Datasets","api":"GET /api/v1/custom-agent-evaluations/datasets","description":"List the project's evaluation datasets","href":"/api-reference/custom-eval/list-datasets"},{"title":"Get Dataset","api":"GET /api/v1/custom-agent-evaluations/datasets/{datasetId}","href":"/api-reference/custom-eval/get-dataset"},{"title":"Create Run","api":"POST /api/v1/custom-agent-evaluations/runs","href":"/api-reference/custom-eval/create-run"},{"title":"Submit Results","api":"POST /api/v1/custom-agent-evaluations/runs/{runId}/results","href":"/api-reference/custom-eval/submit-results"},{"title":"Get Missing Results","api":"GET /api/v1/custom-agent-evaluations/runs/{runId}/missing-results","href":"/api-reference/custom-eval/missing-results"},{"title":"Finalize Run","api":"POST /api/v1/custom-agent-evaluations/runs/{runId}/finalize","href":"/api-reference/custom-eval/finalize-run"},{"title":"Analyze Run","api":"POST /api/v1/custom-agent-evaluations/runs/{runId}/analyze","href":"/api-reference/custom-eval/analyze-run"},{"title":"Get Report","api":"GET /api/v1/custom-agent-evaluations/runs/{runId}/report","href":"/api-reference/custom-eval/get-report"},{"title":"Get Run","api":"GET /api/v1/custom-agent-evaluations/runs/{runId}","href":"/api-reference/custom-eval/get-run"},{"title":"List Runs","api":"GET /api/v1/custom-agent-evaluations/runs","href":"/api-reference/custom-eval/list-runs"},{"title":"List Models","api":"GET /api/v1/agent-monitoring/portability/models","href":"/api-reference/custom-eval/list-models"}]},{"group":"CI Gate","pages":[{"title":"CI/CD Overview","href":"/api-reference/ci-cd/overview"},{"title":"Create CI Run","api":"POST /api/v1/custom-agent-evaluations/runs","href":"/api-reference/ci-cd/create-run"},{"title":"Get Test Cases","api":"GET /api/v1/custom-agent-evaluations/datasets/{datasetId}","href":"/api-reference/ci-cd/get-test-cases"},{"title":"Submit Result","api":"POST /api/v1/custom-agent-evaluations/runs/{runId}/results","href":"/api-reference/ci-cd/submit-result"},{"title":"Finalize CI Run","api":"POST /api/v1/custom-agent-evaluations/runs/{runId}/finalize","href":"/api-reference/ci-cd/finalize-run"},{"title":"Get CI Gate","api":"GET /api/v1/custom-agent-evaluations/runs/{runId}/gate","href":"/api-reference/ci-cd/get-gate"}]},{"group":"Agent Registry","pages":[{"title":"List Agents","api":"GET /api/v1/agents","href":"/api-reference/tracked-agents/list"},{"title":"Register Agent","api":"POST /api/v1/agents","href":"/api-reference/tracked-agents/add"},{"title":"Get Agent","api":"GET /api/v1/agents/{id}","href":"/api-reference/tracked-agents/get"}]}]}]},"basePath":"","title":"Create Dataset - AgentX","deploymentId":"dpl_DyXGnKeCjZyA3sUdfMJ7FjkdBefg","buildId":"dpl_DyXGnKeCjZyA3sUdfMJ7FjkdBefg","dynamics":["c2aa737661","2b790eda92","b0f84f53ff"],"chunks":["OpenApiReferenceProvider-CiUHMMOm.js","snippetPresets-BgfXU7Bw.js","utils-DWvOwgy_.js","style-to-object-BY3JEAdk.js","utils-BN7RXSMJ.js","v4-DIgz8zy6.js","addUniqueKeysToSchema-CFw8bq89.js","conditional-requirements-CFM0id0Q.js","cloneDeep-CAwdfOU2.js","_Uint8Array-BJr8e-ZL.js","isObject-FP1Yky1u.js","ApiExamples-XX0UgMZK.js","usePlaygroundInputsStore-DXDFSa1V.js","shallow-CdX5aREi.js","isEqual-CoOaxexF.js","CodeSnippets-Cg2wpzWm.js","CodeSnippet-D_HwgQO_.js","getApiPlaygroundDisplay-CmCoerhe.js","Page-CVJ-f_k1.js","DynamicMarkdown-CQM9Bhbv.js","SectionHeading-DpbrP8nQ.js","icon-BPCgijHy.js","MethodPill-GQ8DuY0c.js","getPastedJson-Cb5mT31m.js","Check2Icon-a7bUI6e8.js","axios-CsQoIw0U.js"]}