> ## Documentation Index > Fetch the complete documentation index at: https://developers.agentx.so/llms.txt > Use this file to discover all available pages before exploring further. # Custom Evaluations Overview > Run evaluations programmatically against any external agent and score with the LLM judge The Custom Agent Evaluation API runs evaluations against **any agent you own**, regardless of framework: your code drives the agent, submits its outputs, and the engine scores each result synchronously with the LLM judge (plus optional similarity metrics and code scorers). A detailed AI analysis report can be generated after the run, and a [CI gate](/api-reference/ci-cd/get-gate) can turn the finished run into a pass/fail verdict. Examples target a local self-host engine (`http://localhost:4700`). Substitute your own deployment's base URL. The Python SDK wraps this entire surface as `client.evaluations` - see the [Evaluations SDK](/sdk/evaluations/overview). ## When to use this API | Goal | Use | | ------------------------------------------------------------- | ------------------------------------------------------------------------ | | Run a multi-question evaluation and get a score report | This API (or the [Evaluations SDK](/sdk/evaluations/overview)) | | Gate a PR or deploy on eval quality | The same run, then the [CI gate endpoint](/api-reference/ci-cd/get-gate) | | Score one already-ingested trace without re-running the agent | [Evaluate Trace](/api-reference/tracing/evaluate-trace) | | Record traces for browsing | [Submit Trace](/api-reference/tracing/submit-trace) | ## Base path ``` /api/v1/custom-agent-evaluations ``` ## Authentication All routes require a project API key in the `x-api-key` header: ```http theme={null} x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8 ``` ## Lifecycle ``` POST /datasets ← create (or reuse) a dataset with your questions POST /evaluation-settings ← optional: a reusable grading config, independent of any dataset │ POST /runs ← start a run │ (pass evaluationSettingsId to grade with a config above │ instead of the dataset's own criteria) POST /runs/:id/results ← submit batches of agent outputs; each result is judge-scored │ synchronously and every batch returns liveStatistics POST /runs/:id/finalize ← mark the run complete (idempotent; also returns liveStatistics) │ POST /runs/:id/analyze ← run AI analysis (synchronous on self-host) GET /runs/:id/report ← fetch the full report: narrative analysis + statistics GET /runs/:id/gate ← optional: pass/fail verdict for CI ``` ## Limits | Limit | Value | | ---------------------------------- | --------------------------------------------------- | | Max batch size per `/results` call | 10 results | | AI analysis sample | worst 12 + best 5 rated results in the judge prompt | | Judge models per analysis | up to 3 | The judge scores each result synchronously in the `/results` call, so batches of judged results take as long as the underlying LLM calls. Daily judge spend can be capped with `AGENTX_QUOTA_JUDGE_CALLS_PER_DAY` (see [Authentication](/authentication)). ## Endpoints | Method | Path | Description | | -------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/datasets` | [List datasets](/api-reference/custom-eval/list-datasets) | | `POST` | `/datasets` | [Create dataset](/api-reference/custom-eval/create-dataset) | | `GET` | `/datasets/:id` | [Get dataset](/api-reference/custom-eval/get-dataset) | | `DELETE` | `/datasets/:id` | Delete a dataset (`409` if its grading config is attached to a live scorer) | | `POST` | `/datasets/case-preview` | Draft a dataset case from a trace or session (curation step 1) | | `POST` | `/datasets/suggest-expected` | Draft a reference answer for a case on demand (curation step 2) | | `POST` | `/datasets/:id/cases` | Append a curated case to a dataset, with dedupe (curation step 3) | | `GET` | `/evaluation-settings` | List standalone grading configs | | `POST` | `/evaluation-settings` | Create a standalone grading config (criteria, judge prompt/model, similarity metrics, code scorers), reusable across datasets | | `GET` | `/evaluation-settings/:id` | Get a grading config | | `POST` | `/runs` | [Create run](/api-reference/custom-eval/create-run) | | `GET` | `/runs` | [List runs](/api-reference/custom-eval/list-runs) | | `GET` | `/runs/:id` | [Get run](/api-reference/custom-eval/get-run) | | `POST` | `/runs/:id/results` | [Submit results](/api-reference/custom-eval/submit-results) | | `GET` | `/runs/:id/missing-results` | [Resume support: submitted idempotency keys](/api-reference/custom-eval/missing-results) | | `POST` | `/runs/:id/finalize` | [Finalize run](/api-reference/custom-eval/finalize-run) | | `POST` | `/runs/:id/analyze` | [Analyze run](/api-reference/custom-eval/analyze-run) | | `GET` | `/runs/:id/analyze-status` | Analysis status (terminal immediately on self-host; see [Analyze Run](/api-reference/custom-eval/analyze-run)) | | `GET` | `/runs/:id/report` | [Get report](/api-reference/custom-eval/get-report) | | `GET` | `/runs/:id/gate` | [CI gate: pass/fail the run](/api-reference/ci-cd/get-gate) | | `POST` | `/prompts` | Create a prompt in the prompt registry | | `GET` | `/prompts` | List prompts | | `GET` | `/prompts/:identifier` | Get a prompt by name or id (`?version=` for a specific version) | The judge model catalog lives on a separate route - see [List Models](/api-reference/custom-eval/list-models). All endpoints share the engine's global error envelope for authentication (`401`) and rate limiting (`429`) - see [Errors](/errors). The live-monitoring surface (patterns, online evaluators, scorer groups, signals) is documented conceptually under the Scorers and Online Evaluation tabs and through the [Python SDK](/sdk/monitor) rather than as HTTP reference pages.