Custom Evaluations Overview

The Custom Agent Evaluation API runs evaluations against any agent you own, regardless of framework: your code drives the agent, submits its outputs, and the engine scores each result synchronously with the LLM judge (plus optional similarity metrics and code scorers). A detailed AI analysis report can be generated after the run, and a CI gate can turn the finished run into a pass/fail verdict.

Examples target a local self-host engine (http://localhost:4700). Substitute your own deployment’s base URL. The Python SDK wraps this entire surface as client.evaluations - see the Evaluations SDK.

When to use this API

Goal	Use
Run a multi-question evaluation and get a score report	This API (or the Evaluations SDK)
Gate a PR or deploy on eval quality	The same run, then the CI gate endpoint
Score one already-ingested trace without re-running the agent	Evaluate Trace
Record traces for browsing	Submit Trace

Base path

/api/v1/custom-agent-evaluations

Authentication

All routes require a project API key in the x-api-key header:

x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8

Lifecycle

POST /datasets              ← create (or reuse) a dataset with your questions
POST /evaluation-settings   ← optional: a reusable grading config, independent of any dataset
        │
POST /runs                  ← start a run
        │                     (pass evaluationSettingsId to grade with a config above
        │                      instead of the dataset's own criteria)
POST /runs/:id/results      ← submit batches of agent outputs; each result is judge-scored
        │                     synchronously and every batch returns liveStatistics
POST /runs/:id/finalize     ← mark the run complete (idempotent; also returns liveStatistics)
        │
POST /runs/:id/analyze      ← run AI analysis (synchronous on self-host)
GET  /runs/:id/report       ← fetch the full report: narrative analysis + statistics
GET  /runs/:id/gate         ← optional: pass/fail verdict for CI

Limits

Limit	Value
Max batch size per `/results` call	10 results
AI analysis sample	worst 12 + best 5 rated results in the judge prompt
Judge models per analysis	up to 3

The judge scores each result synchronously in the /results call, so batches of judged results take as long as the underlying LLM calls. Daily judge spend can be capped with AGENTX_QUOTA_JUDGE_CALLS_PER_DAY (see Authentication).

Endpoints

Method	Path	Description
`GET`	`/datasets`	List datasets
`POST`	`/datasets`	Create dataset
`GET`	`/datasets/:id`	Get dataset
`DELETE`	`/datasets/:id`	Delete a dataset (`409` if its grading config is attached to a live scorer)
`POST`	`/datasets/case-preview`	Draft a dataset case from a trace or session (curation step 1)
`POST`	`/datasets/suggest-expected`	Draft a reference answer for a case on demand (curation step 2)
`POST`	`/datasets/:id/cases`	Append a curated case to a dataset, with dedupe (curation step 3)
`GET`	`/evaluation-settings`	List standalone grading configs
`POST`	`/evaluation-settings`	Create a standalone grading config (criteria, judge prompt/model, similarity metrics, code scorers), reusable across datasets
`GET`	`/evaluation-settings/:id`	Get a grading config
`POST`	`/runs`	Create run
`GET`	`/runs`	List runs
`GET`	`/runs/:id`	Get run
`POST`	`/runs/:id/results`	Submit results
`GET`	`/runs/:id/missing-results`	Resume support: submitted idempotency keys
`POST`	`/runs/:id/finalize`	Finalize run
`POST`	`/runs/:id/analyze`	Analyze run
`GET`	`/runs/:id/analyze-status`	Analysis status (terminal immediately on self-host; see Analyze Run)
`GET`	`/runs/:id/report`	Get report
`GET`	`/runs/:id/gate`	CI gate: pass/fail the run
`POST`	`/prompts`	Create a prompt in the prompt registry
`GET`	`/prompts`	List prompts
`GET`	`/prompts/:identifier`	Get a prompt by name or id (`?version=` for a specific version)

The judge model catalog lives on a separate route - see List Models. All endpoints share the engine’s global error envelope for authentication (401) and rate limiting (429) - see Errors. The live-monitoring surface (patterns, online evaluators, scorer groups, signals) is documented conceptually under the Scorers and Online Evaluation tabs and through the Python SDK rather than as HTTP reference pages.

Tracing