# Create CI Run
Source: https://developers.agentx.so/api-reference/ci-cd/create-run
POST /api/v1/custom-agent-evaluations/runs
Start the evaluation run a CI pipeline will gate on
A CI run **is** a standard evaluation run - this is the same endpoint as
[Create Run](/api-reference/custom-eval/create-run), shown here in its CI framing. Create the
run at the start of the pipeline job, drive your agent through the dataset's cases, submit and
finalize, then compute the [gate](/api-reference/ci-cd/get-run).
Two fields matter most in CI:
* **`evaluationSubject.version`** (or `evaluationSubject.metadata.version`): tag the run with
the branch or commit under test, so version-to-version comparisons in the dashboard line up
with your git history.
* **`split`**: run only the dataset cases tagged with a named split (e.g. `"smoke"`) for fast
PR gates, keeping the full dataset for nightly runs.
## Authentication
Project API key (use a CI secret, never source control).
## Body
Dataset ID to evaluate against.
Metadata about the agent under test. In CI, include a `version` tag:
```json theme={null}
{
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"version": "feat/new-retrieval@a1b2c3d4"
}
```
Grade with a standalone grading config instead of the dataset's own criteria.
How the run was triggered.
Named case subset to cover (cases tagged via `main_question.splits`). The caller filters
which cases it executes; the split is recorded on the run's subject.
## Response
Returns `201 Created` with `{ runId, datasetId, status: "in_progress", smokeTestVariants }` -
see [Create Run](/api-reference/custom-eval/create-run) for the full field reference. Read the
cases to execute from the dataset itself
([Get Test Cases](/api-reference/ci-cd/get-test-cases)).
## Errors
| Status | Body | Meaning |
| ------ | -------------------------------------- | -------------------------------------- |
| `400` | `{ "error": "datasetId is required" }` | Missing or non-string `datasetId` |
| `404` | `{ "error": "Dataset not found" }` | No dataset with this id in the project |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"version": "feat/new-retrieval@a1b2c3d4"
},
"split": "smoke"
}'
```
```python Python SDK theme={null}
import sys
report = (
client.evaluations.run(
"dS4tG7hNb2VxZ8kQ5wMyA",
subject={"kind": "custom_agent", "displayName": "Customer Support Bot",
"metadata": {"version": "feat/new-retrieval@a1b2c3d4"}},
split="smoke",
)
.execute(my_agent) # called once per case
.finalize()
)
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
```
```json 201 Created theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
```
# Finalize CI Run
Source: https://developers.agentx.so/api-reference/ci-cd/finalize-run
POST /api/v1/custom-agent-evaluations/runs/{runId}/finalize
Close the run before computing the CI gate
Marks the run `"completed"` and returns the final rating aggregate. Finalize after every result
batch has been submitted and **before** calling the [gate](/api-reference/ci-cd/get-run) - the
gate reads whatever ratings are stored, and finalizing pins the run so nothing lands after the
verdict. This is the same endpoint as
[Finalize Run](/api-reference/custom-eval/finalize-run); see that page for the full reference.
Finalizing is idempotent, so a retried pipeline step is harmless: a second call returns the
same `"completed"` response, and a `"failed"` run stays failed (returned as
`status: "failed"`, not an error).
## Authentication
Project API key.
## Path Parameters
Run ID.
## Response
Run ID.
`"completed"` (or `"failed"` if the run had previously failed).
Final aggregate: `{ averageRating, minRating, maxRating, ratedCount, skippedCount,
failedCount }`. A nonzero `skippedCount` in CI usually means the judge could not score
(e.g. missing judge provider key) - the gate would then fail with "no rated results", so
check this before gating.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------ | ---------------------------------- |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/finalize \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```json 200 OK theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"liveStatistics": {
"averageRating": 7.8,
"minRating": 5,
"maxRating": 10,
"ratedCount": 8,
"skippedCount": 0,
"failedCount": 0
}
}
```
# Get CI Gate
Source: https://developers.agentx.so/api-reference/ci-cd/get-run
GET /api/v1/custom-agent-evaluations/runs/{runId}/gate
Compute the pass/fail verdict for a finalized run
Computes the CI gate for a run: an absolute rating floor, a no-regression check against the
dataset's previous completed run, or both. The verdict is computed fresh from the run's stored
ratings on every call, so re-running a failed CI job re-evaluates against current state. Exit
your pipeline non-zero when `passed` is `false`.
To poll a run's progress or read its per-result scores, use
[Get Run](/api-reference/custom-eval/get-run) instead; this endpoint is the verdict.
## Authentication
Project API key.
## Path Parameters
Run ID of a [finalized](/api-reference/ci-cd/finalize-run) run.
## Query Parameters
At least one of `failUnder` and `noRegression` is required.
Absolute floor, 0-10: the check fails when the run's average rating is below this value (or
when the run has no rated results).
`true` (or `1`): fail when the average dropped more than `tolerance` below the dataset's
previous completed rated run. Passes explicitly when no baseline exists. Trace evaluations
(`runSource: "trace-eval"`) are never used as the baseline.
Allowed drop for the no-regression check. Judge scores are noisy; an exact comparison would
flake builds on variance rather than regressions.
`true` (or `1`): persist this verdict into the project's gate history (the dashboard's CI
page). A recorded failed gate also fires the webhook channels configured on the project's
monitoring profiles. The Python SDK's `report.gate()` records by default; ad hoc previews
should omit it.
Label stored with a recorded verdict (max 60 characters), e.g. `"github-actions"` or
`"pytest"`.
Gate on a named additional judge scorer instead of the primary (multi-judge runs, self-host):
pass the scorer's id or name, e.g. `scorer=Safety`. `failUnder` then checks that scorer's own
average, and `noRegression` compares against the previous run's average for the same scorer.
The response echoes it back as `gatedScorer: { id, name }` (`null` when gating the primary).
An unknown id or name is a `400`, never a silently-passing gate.
## Response
Run ID.
Dataset the run scored against.
The run's average rating, rounded to 2 decimals. `null` when nothing was rated.
Total submitted results.
The previous completed rated run used as the no-regression baseline, or `null` (check not
requested, or no baseline exists).
The baseline run's average rating.
`{ id, name }` of the additional scorer being gated when `?scorer=` was passed; `null` when
the gate ran against the primary scorer. When set, `averageRating`, `baselineAverage`, and
every check use that scorer's own per-result verdicts.
One entry per requested check:
`[{ "check": "fail-under" | "no-regression", "passed": boolean, "threshold": number | null, "actual": number | null, "detail": string }]`.
`detail` is a CI-log-friendly sentence explaining the outcome.
`true` only when every requested check passed. This is the field to exit on.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------------------------------------------------------------------ | ---------------------------------------------- |
| `400` | `{ "error": "At least one check is required: failUnder=<0-10> and/or noRegression=true" }` | Neither check requested |
| `400` | `{ "error": "failUnder must be a number" }` | Non-numeric `failUnder` (same for `tolerance`) |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
import sys
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
gate = report.gate(fail_under=7, no_regression=True, caller="github-actions")
if not gate.passed:
sys.exit(1)
```
```python pytest theme={null}
from agentx.testing import assert_evaluation
def test_support_agent_quality():
report = client.evaluations.run(dataset_id, subject).execute(my_agent).finalize()
# Same gate in pytest ergonomics: raises EvaluationAssertionError on failure
# (recorded in gate history with caller="pytest").
assert_evaluation(report, min_rating=7, no_regression=True)
```
```json 200 OK (passed) theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"averageRating": 7.8,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": true,
"threshold": 7,
"actual": 7.8,
"detail": "Average rating 7.8 >= floor 7"
},
{
"check": "no-regression",
"passed": true,
"threshold": 7.5,
"actual": 7.8,
"detail": "Average rating 7.8 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": true
}
```
```json 200 OK (failed) theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"averageRating": 5.2,
"resultCount": 8,
"baselineRunId": "rB2sN6wQj4XcV9kM3pLtA",
"baselineAverage": 7.5,
"checks": [
{
"check": "fail-under",
"passed": false,
"threshold": 7,
"actual": 5.2,
"detail": "Average rating 5.2 < floor 7"
},
{
"check": "no-regression",
"passed": false,
"threshold": 7.5,
"actual": 5.2,
"detail": "Average rating 5.2 vs previous run's 7.5 (tolerance 0.5)"
}
],
"passed": false
}
```
```json 400 No check requested theme={null}
{
"error": "At least one check is required: failUnder=<0-10> and/or noRegression=true"
}
```
# Get Test Cases
Source: https://developers.agentx.so/api-reference/ci-cd/get-test-cases
GET /api/v1/custom-agent-evaluations/datasets/{datasetId}
Read the dataset's questions to drive a CI evaluation
The test cases a CI run executes are the dataset's own `questions` array - there is no separate
test-case endpoint. Fetch the dataset, iterate `questions`, run your agent on each
`main_question.query`, and submit each answer with the case's array position as
`questionIndex`.
This is the same endpoint as [Get Dataset](/api-reference/custom-eval/get-dataset); this page
covers the CI angle.
## Authentication
Project API key.
## Path Parameters
Dataset `_id`.
## Using the response in CI
For each element of `questions` (0-based index `i`):
* `main_question.query` is the input to pose to your agent.
* Submit the answer with `questionIndex: i` (and `runNumber` 1 through `numberOfRequests` when
repeating cases for consistency).
* `main_question.splits`, when present, tags the case's named subsets - a run created with
`"split": "smoke"` should execute only cases tagged `"smoke"`, keeping their original
indexes.
* `expectedResults`, `expectedTrajectory`, and the other grading fields are read server-side
by the judge and scorers; your harness doesn't need them (though `expectedResults` is useful
for local debugging).
## Errors
| Status | Body | Meaning |
| ------ | ---------------------------------- | -------------------------------------- |
| `404` | `{ "error": "Dataset not found" }` | No dataset with this id in the project |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```bash Extract just the queries (jq) theme={null}
curl -s "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: $AGENTX_API_KEY" \
| jq -r '.questions[].main_question.query'
```
```json 200 OK theme={null}
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"numberOfRequests": 1,
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen.", "splits": ["smoke"] } },
{ "main_question": { "query": "What payment methods do you accept?" } },
{ "main_question": { "query": "How long does shipping take?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
```
# CI/CD Overview
Source: https://developers.agentx.so/api-reference/ci-cd/overview
Gate merges and deploys on evaluation quality with the run gate endpoint
CI/CD evaluation on AgentX is an ordinary
[custom evaluation run](/api-reference/custom-eval/overview) plus one extra call: after the run
finalizes, `GET /runs/:id/gate` turns its ratings into a binary **passed / failed** verdict
your pipeline can exit on. Your agent runs in your infrastructure; the engine scores results
and computes the gate.
There is no separate CI run type or `/ci-runs` endpoint family on the self-host engine - a
CI run is a normal evaluation run, so it appears in Evaluate > Runs with full results,
analysis, and history like any other. (The hosted platform additionally offers a lightweight
CI Ingest API, which the SDK's `run_eval()` wraps - see [SDK CI/CD](/sdk/ci-cd).) The
[Python SDK](/sdk/ci-cd) wraps the whole self-host flow too, including `report.gate(...)`
and the pytest helper `agentx.testing.assert_evaluation`.
## How it works
```
CI pipeline
│
├─ POST /custom-agent-evaluations/runs ← create the run
│
├─ (run your agent per dataset case)
│
├─ POST /custom-agent-evaluations/runs/:id/results ← submit + judge-score each batch
│
├─ POST /custom-agent-evaluations/runs/:id/finalize ← close the run
│
└─ GET /custom-agent-evaluations/runs/:id/gate ← verdict: passed true/false
```
## Gate checks
The gate runs one or both of these checks (at least one is required) and passes only when
**every requested check passes**:
1. **Absolute floor** (`failUnder=<0-10>`): the run's average rating must be at or above the
floor. A run with no rated results fails this check.
2. **No regression** (`noRegression=true`): the average must not have dropped more than
`tolerance` (default `0.5`) below the dataset's previous completed run. Judge scores are
noisy, so an exact comparison would flake builds on variance; tune `tolerance` to your
dataset. With no prior completed rated run, the check passes explicitly ("nothing to
regress against"). Single-trace evaluations are never used as the baseline.
The verdict is computed fresh from stored ratings on every call, so re-running a failed CI job
re-evaluates against current state.
## Gate history and alerting
`record=true` appends the verdict to the project's gate history - what the dashboard's CI page
lists - along with an optional `caller` label (e.g. `"github-actions"`). A recorded **failed**
gate also fires the webhook channels configured on the project's monitoring profiles, so a
gate blocking a merge can reach Slack. Preview calls omit `record` and stay compute-only.
## Smoke splits
Tag a subset of dataset cases with a named split (`main_question.splits: ["smoke"]`) and create
the run with `"split": "smoke"` to gate PRs on a fast subset while nightly jobs run the full
dataset. Case indexes are preserved, so per-case comparisons line up across split and full
runs.
## Endpoints
| Method | Path | Description |
| ------ | --------------------------------------------- | -------------------------------------------------------------------- |
| `POST` | `/custom-agent-evaluations/runs` | [Create the run](/api-reference/ci-cd/create-run) |
| `GET` | `/custom-agent-evaluations/datasets/:id` | [Read the dataset's test cases](/api-reference/ci-cd/get-test-cases) |
| `POST` | `/custom-agent-evaluations/runs/:id/results` | [Submit results](/api-reference/ci-cd/submit-result) |
| `POST` | `/custom-agent-evaluations/runs/:id/finalize` | [Finalize the run](/api-reference/ci-cd/finalize-run) |
| `GET` | `/custom-agent-evaluations/runs/:id/gate` | [Compute the gate verdict](/api-reference/ci-cd/get-run) |
## Minimal pipeline example
```bash theme={null}
BASE=http://localhost:4700/api/v1/custom-agent-evaluations
KEY="x-api-key: $AGENTX_API_KEY"
RUN_ID=$(curl -s -X POST "$BASE/runs" -H "$KEY" -H "Content-Type: application/json" \
-d '{"datasetId":"'"$DATASET_ID"'","runSource":"sdk"}' | jq -r .runId)
# ... run your agent per case and POST batches to $BASE/runs/$RUN_ID/results ...
curl -s -X POST "$BASE/runs/$RUN_ID/finalize" -H "$KEY" > /dev/null
PASSED=$(curl -s "$BASE/runs/$RUN_ID/gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
-H "$KEY" | jq -r .passed)
[ "$PASSED" = "true" ] || { echo "CI gate FAILED"; exit 1; }
```
# Submit Result
Source: https://developers.agentx.so/api-reference/ci-cd/submit-result
POST /api/v1/custom-agent-evaluations/runs/{runId}/results
Submit and score CI case results as the pipeline produces them
Submits a batch of up to 10 case results for the CI run; each new result is judge-scored
synchronously inside the request. This is the same endpoint as
[Submit Results](/api-reference/custom-eval/submit-results) - see that page for the full
result-object reference. This page covers the CI angle.
## CI-relevant behavior
* **Idempotency makes retries safe.** Give every result an `idempotencyKey`
(`{runId}:{caseId}:run-{runNumber}`); a retried or duplicated batch returns the stored
scores instead of re-scoring, so a flaky network step never double-bills judge calls. On
job restart, fetch the already-submitted keys from
[Get Missing Results](/api-reference/custom-eval/missing-results) and skip them.
* **Failures are results too.** A case where your agent threw should be submitted with
`error: { "type", "message" }` - it scores 0 and drags the average down honestly, instead
of silently shrinking the run.
* **Watch the average as you go.** Every batch response carries `liveStatistics`; a pipeline
can log progress or bail out early when the average is already hopeless. There is no
server-side fail-fast: stopping early is the client's decision, and the
[gate](/api-reference/ci-cd/get-run) computes over whatever was submitted.
* **Generous timeouts.** Scoring is a judge LLM call per result; the Python SDK uses a long
per-batch timeout and disables transport retries for exactly this call.
## Errors
| Status | Body | Meaning |
| ------ | --------------------------------------------------- | --------------------------------------- |
| `400` | `{ "error": "batchId is required" }` | Missing `batchId` |
| `400` | `{ "error": "results must be a non-empty array" }` | Missing or empty `results` |
| `400` | `{ "error": "Batch size must not exceed 10" }` | More than 10 results in one call |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
| `409` | `{ "error": "Run is already in a terminal state" }` | The run was already finalized or failed |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "ci-batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login page to reset your password." },
"timings": { "latencyMs": 1340 }
}
]
}'
```
```json 200 OK theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "ci-batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly described the password reset flow.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": null,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
```
```json 409 Already finalized theme={null}
{
"error": "Run is already in a terminal state"
}
```
# Analyze Run
Source: https://developers.agentx.so/api-reference/custom-eval/analyze-run
POST /api/v1/custom-agent-evaluations/runs/{runId}/analyze
Run AI analysis over a run's results and store the structured report
Runs an LLM analysis over the run's scored results and stores a structured narrative report:
overall summary, consistency, instruction adherence, response patterns, reasoning quality, tool
usage, and prioritized recommendations. Fetch the stored report with
[Get Report](/api-reference/custom-eval/get-report).
The analysis prompt samples the **worst 12 and best 5** rated results (all of them when the run
has 17 or fewer), and each sampled result is independently re-scored by up to 3 judge models to
produce per-item agreement evidence.
On the self-host engine this endpoint is **synchronous**: it returns only once the judges are
done, so the response status is already terminal (`"completed"` or `"failed"`), and the first
`GET /runs/:runId/analyze-status` poll reads that same terminal status. The `mode: "sync"`
field says so on the wire. (Hosted AgentX queues a durable job and returns `"pending"`
instead; the SDK's poll loop handles both.) Re-running analysis overwrites the stored row.
## Authentication
Project API key.
## Path Parameters
Run ID. Analyze after [finalizing](/api-reference/custom-eval/finalize-run), so the analysis
covers every result.
## Body
Judge models to score the evidence sample with, up to 3: `[{ "model": string }]`. Model ids
come from the [model catalog](/api-reference/custom-eval/list-models). The first entry also
writes the narrative. Omit to use the default judge model alone.
`"balanced"` or `"quality_first"`. Echoed back; recorded with the analysis request.
## Response
The run ID (analysis is keyed by run).
Same value as `evaluationId`, kept for SDK compatibility with the hosted job queue.
`"completed"`, or `"failed"` (for example, when the run has no scored results yet, or the
judge model returned no analysis - the failure reason is readable from
`GET /runs/:runId/analyze-status`).
Always `"sync"` on self-host.
Echoed back.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------ | ---------------------------------- |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
## Checking status
`GET /api/v1/custom-agent-evaluations/runs/{runId}/analyze-status` returns
`{ evaluationId, jobId, status, progress, failureReason, warnings, cost, etaUpdatedAt, overflowStats }`.
`status` is `"not_started"` before any analysis, otherwise the stored row's terminal
`"completed"` or `"failed"`; a failure carries
`failureReason: { code: "ANALYSIS_FAILED", message, retryable: true }`. The progress/cost/
overflow fields exist for wire compatibility with the hosted job queue and are static on
self-host.
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/analyze \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "judges": [{ "model": "gpt-5.6-luna" }, { "model": "claude-opus-4-7" }] }'
```
```python Python SDK theme={null}
client.evaluations.analyze_run("rK7dP2qWx9TzB4mV6nJcE")
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
```
```json 200 OK theme={null}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"mode": "sync",
"qualityMode": "balanced"
}
```
```json 200 OK (nothing to analyze) theme={null}
{
"evaluationId": "rK7dP2qWx9TzB4mV6nJcE",
"jobId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "failed",
"mode": "sync",
"qualityMode": "balanced"
}
```
```json 404 Not found theme={null}
{
"error": "Run not found"
}
```
# Create Dataset
Source: https://developers.agentx.so/api-reference/custom-eval/create-dataset
POST /api/v1/custom-agent-evaluations/datasets
Create an evaluation dataset with questions and scoring configuration
Creates a dataset: the questions your agent will be evaluated against plus the grading
configuration (judge criteria, similarity metrics, code scorers) used when a run doesn't bring
its own [evaluation-settings config](/api-reference/custom-eval/overview#endpoints). Creation
also seeds version `v0` of the dataset's version history.
## Authentication
Project API key.
## Body
Dataset display name.
Question objects. Not validated at creation time (an empty or malformed array is accepted),
but a run against a dataset with no usable questions has nothing to score. Each question
wraps its fields in `main_question`:
```json theme={null}
[
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "The user should click Forgot Password on the login screen."
}
}
]
```
| Field | Type | Description |
| -------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `query` | string | The question posed to the agent |
| `expectedResults` | string | Reference answer. Enables similarity scoring and is shown to the judge; required per-case by `requiresExpected` grading configs |
| `judgeGuideline` | string | Case-specific guidance added to the judge prompt |
| `retrievalContext` | string | Statically pinned context for `{context}`-referencing judge prompts (used only when the result and its linked trace carry none) |
| `expectedRetrievalContext` | string \| string\[] | Reference retrieval: what a correct retriever should have fetched. Compared to the actually-retrieved context by token Jaccard similarity - no judge call |
| `expectedTrajectory` | object | `{ "tools": string[], "mode": "strict" \| "unordered" \| "subset" \| "superset" }` - expected tool calls, matched against the linked trace's actual tool sequence |
| `smokeTest` | object | `{ "enabled": boolean, "count": number, "guidance": string }` - generate paraphrase variants of this question at run creation for consistency testing |
| `splits` | string\[] | Named subset tags; a run created with `split` covers only cases tagged with it |
Human-readable description.
How many times each question is run per evaluation (for consistency testing).
What a good response looks like. Included in the LLM scoring prompt.
What a bad response looks like. Included in the LLM scoring prompt.
Scoring rubric. Included in the LLM scoring prompt.
Enable cosine similarity scoring against `expectedResults` (requires an embeddings-capable
provider key). Only `{ "enabled": true }` activates it; `model` optionally selects the
embedding model.
```json theme={null}
{ "enabled": true, "model": "text-embedding-3-small" }
```
Enable Jaccard token-overlap scoring against `expectedResults`: `{ "enabled": true }`.
Enable BLEU scoring against `expectedResults`: `{ "enabled": true }`.
Enable ROUGE scoring against `expectedResults`: `{ "enabled": true }`.
Deterministic code scorers run on every result. Each:
`{ "id"?: string, "name": string, "code": string, "enabled"?: boolean }`. Entries with empty
`code` are dropped; `enabled` defaults to `true`.
## Response
Returns `201 Created` with the full dataset document, keyed by `_id`. `status` is always
`"published"` on self-host. The enabled similarity metrics come back as top-level keys
(`vectorSimilarity`, `jaccardSimilarity`, ...), matching what was sent.
## Errors
| Status | Body | Meaning |
| ------ | --------------------------------- | -------------------------------------- |
| `400` | `{ "error": "name is required" }` | `name` missing, not a string, or blank |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/datasets \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "Click Forgot Password on the login screen."
}
},
{
"main_question": {
"query": "What payment methods do you accept?"
}
}
],
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true }
}'
```
```python Python SDK theme={null}
dataset = (
client.evaluations.datasets.builder(
"Customer Support Q3 2026",
description="Core support questions",
number_of_requests=3,
acceptance_criteria="Accurate, empathetic, resolves the issue",
rejection_criteria="Hallucinates, ignores the question",
vector_similarity=True,
jaccard_similarity=True,
)
.add_case("How do I reset my password?",
expected_results="Click Forgot Password on the login screen.")
.add_case("What payment methods do you accept?")
.publish()
)
```
```json 201 Created theme={null}
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true },
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } },
{ "main_question": { "query": "What payment methods do you accept?" } }
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
```
```json 400 Missing name theme={null}
{
"error": "name is required"
}
```
# Create Run
Source: https://developers.agentx.so/api-reference/custom-eval/create-run
POST /api/v1/custom-agent-evaluations/runs
Start a new evaluation run against a dataset
Initialises a new evaluation run with status `"in_progress"`. Submit your agent's outputs to it
with [Submit Results](/api-reference/custom-eval/submit-results), then
[finalize](/api-reference/custom-eval/finalize-run). If any dataset case has smoke testing
enabled, the paraphrase variants are generated once here and frozen for the run's lifetime.
## Authentication
Project API key.
## Body
Dataset ID to evaluate against.
Grade with a standalone, reusable grading config (criteria, judge prompt/model, similarity
metrics, code scorers) instead of the dataset's own configuration. Lets one config run
against any dataset. Omit to use the dataset's own criteria.
Grade with a [scorer group](/monitor/scorer-groups) - the group's weighted aggregate fills
the run's rating, judge members land as per-row verdicts, and deterministic members as scorer
rows. A group is a complete grading recipe: when it resolves, `evaluationSettingsId` and
`additionalScorerIds` are ignored.
Extra LLM judge scorer ids (up to 4, self-host) that also grade every result - one agent
execution, N verdicts. The primary scorer keeps the `rating` column; each extra verdict lands
in the result row's `judgeScorerResults` and rolls up into the run's `scorerBreakdown`
([Get Run](/api-reference/custom-eval/get-run)). Duplicates of the primary are dropped.
Free-form metadata about the agent being evaluated (e.g. `kind`, `displayName`, `framework`,
`runtime`, `agentInstructions`). Stored on the run and echoed back by
[Get Run](/api-reference/custom-eval/get-run). A `version` field (or
`metadata.version`) tags the run for version-to-version comparison in the dashboard.
How the run was triggered, e.g. `"sdk"`. Trace evaluations use `"trace-eval"` internally and
are excluded from CI-gate baselines.
SDK name/version info, stored for diagnostics.
Named case subset: run only the dataset cases tagged with this split
(`main_question.splits`). The caller does the filtering when executing; the split name is
recorded on the run's subject so split runs are visibly split runs. Original question
indexes are preserved, so per-case comparisons line up with full runs.
## Response
Returns `201 Created`.
Use in all subsequent calls for this run.
Echoed back.
Always `"in_progress"` on creation.
One group per dataset case with `smokeTest.enabled`:
`[{ "questionIndex": number, "variants": string[] }]`. `null` when no case requests smoke
testing. Run each variant like a normal case and submit its result with
`isSmokeTestVariant: true`.
## Errors
| Status | Body | Meaning |
| ------ | -------------------------------------- | -------------------------------------- |
| `400` | `{ "error": "datasetId is required" }` | Missing or non-string `datasetId` |
| `404` | `{ "error": "Dataset not found" }` | No dataset with this id in the project |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"framework": "langchain",
"version": "v2-retrieval"
},
"runSource": "sdk"
}'
```
```python Python SDK theme={null}
from agentx.evaluations.models import EvaluationSubject
# client.evaluations.run(dataset_id, subject, ...) drives the whole lifecycle;
# init_run alone creates the run to drive manually.
run = client.evaluations.init_run(
dataset_id="dS4tG7hNb2VxZ8kQ5wMyA",
subject=EvaluationSubject(kind="custom_agent", display_name="Customer Support Bot"),
)
```
```json 201 Created theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "in_progress",
"smokeTestVariants": null
}
```
```json 404 Dataset not found theme={null}
{
"error": "Dataset not found"
}
```
# Finalize Run
Source: https://developers.agentx.so/api-reference/custom-eval/finalize-run
POST /api/v1/custom-agent-evaluations/runs/{runId}/finalize
Mark a run as complete and get the final rating statistics
Marks the run `"completed"` and returns the authoritative rating aggregate, recomputed from
every stored result. After finalizing, no more results are accepted (further `/results` calls
return `409`), and the run is ready for
[Analyze Run](/api-reference/custom-eval/analyze-run) or the
[CI gate](/api-reference/ci-cd/get-run).
Finalizing is **idempotent**: calling it again on a completed run returns the same
`"completed"` response. Finalizing a `"failed"` run returns `status: "failed"` (with its
statistics) rather than flipping it to completed - and rather than an error, so retried
finalize calls never crash a pipeline. You can finalize with partial results; use
[Get Missing Results](/api-reference/custom-eval/missing-results) first to check coverage.
## Authentication
Project API key.
## Path Parameters
Run ID.
## Body
No body required.
## Response
Run ID.
`"completed"`, or `"failed"` if the run had previously failed.
Final rating aggregate, recomputed from every stored result:
`{ averageRating, minRating, maxRating, ratedCount, skippedCount, failedCount }`. Same shape
as [Submit Results](/api-reference/custom-eval/submit-results)' `liveStatistics`. Available
without calling [Analyze Run](/api-reference/custom-eval/analyze-run); analysis only adds the
LLM-written qualitative report on top.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------ | ---------------------------------- |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/finalize \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
result = client.evaluations.finalize_run("rK7dP2qWx9TzB4mV6nJcE")
print(result["liveStatistics"]["averageRating"])
```
```json 200 OK theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"status": "completed",
"liveStatistics": {
"averageRating": 7.8,
"minRating": 5,
"maxRating": 10,
"ratedCount": 6,
"skippedCount": 0,
"failedCount": 0
}
}
```
```json 404 Not found theme={null}
{
"error": "Run not found"
}
```
# Get Dataset
Source: https://developers.agentx.so/api-reference/custom-eval/get-dataset
GET /api/v1/custom-agent-evaluations/datasets/{datasetId}
Fetch a single evaluation dataset by id
Returns one dataset from the project selected by your API key, including its full `questions`
array and grading configuration. Same document shape as the items in
[List Datasets](/api-reference/custom-eval/list-datasets).
## Authentication
Project API key.
## Path Parameters
Dataset `_id`.
## Errors
| Status | Body | Meaning |
| ------ | ---------------------------------- | -------------------------------------- |
| `404` | `{ "error": "Dataset not found" }` | No dataset with this id in the project |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets/dS4tG7hNb2VxZ8kQ5wMyA" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
dataset = client.evaluations.datasets.get("dS4tG7hNb2VxZ8kQ5wMyA")
```
```json 200 OK theme={null}
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions",
"numberOfRequests": 3,
"vectorSimilarity": { "enabled": true },
"jaccardSimilarity": { "enabled": true },
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question",
"questions": [
{
"main_question": {
"query": "How do I reset my password?",
"expectedResults": "Click Forgot Password on the login screen."
}
}
],
"status": "published",
"createdAt": "2026-08-27T10:00:00.000Z"
}
```
```json 404 Not found theme={null}
{
"error": "Dataset not found"
}
```
# Get Report
Source: https://developers.agentx.so/api-reference/custom-eval/get-report
GET /api/v1/custom-agent-evaluations/runs/{runId}/report
Get the stored analysis report with statistics and low-scoring cases
Returns the report [Analyze Run](/api-reference/custom-eval/analyze-run) stored for this run:
the narrative analysis hoisted to the top level, plus the run's identifiers, the statistics
computed when the analysis ran, and the run's low-scoring cases. Requires an analysis to exist -
a run that was never analyzed returns `404`, so an empty report is never mistaken for a run
that scored nothing.
## Authentication
Project API key.
## Path Parameters
Run ID.
## Response
Narrative analysis fields (top level, from the analysis body; absent when the analysis
failed):
| Field | Type | Description |
| ---------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `summary` | string | A few sentences on how the agent performed overall |
| `consistencyScore` | number | 0-10: how consistent responses were across similar inputs |
| `instructionAdherence` | object | `{ score (0-10), analysis, deviations[], rating }` |
| `responsePatterns` | object | `{ similarities[], differences[], outliers[], rating }` |
| `reasoningAnalysis` | object | `{ cotQuality, reasoningPatterns[], reasoningGaps[], rating }` |
| `toolUsageAnalysis` | object | `{ effectiveness, patterns[], issues[], rating }` |
| `recommendations` | array | `[{ category, priority, recommendation, reasoning }]` - `category` is one of `"instructions"`, `"tools"`, `"knowledge"`, `"reasoning"`, `"consistency"`, `"other"`; `priority` and every `rating` above are `"high"` \| `"medium"` \| `"low"` |
| `instructionChanges` | array | Always `[]` for external agents (the engine doesn't own the agent's instructions) |
| `overallAssessment` | object | `{ strengths[], weaknesses[], rating }` |
Run fields (always present):
Run ID.
Dataset the run scored against.
The stored analysis status: `"completed"` or `"failed"`.
Computed when the analysis ran:
`{ numberOfRuns, averageRating, minRating, maxRating, ratingVariance }`.
Results with a rating of 5 or below (smoke-test variants excluded), derived fresh from the
run's own rows: `[{ query, response, rating, justification }]`.
Similarity aggregates (cosine/Jaccard/BLEU/ROUGE) are per-result, not report-level - read
them from [Get Run](/api-reference/custom-eval/get-run)'s `results` rows. `strengths`,
`weaknesses`, and the overall rating live under `overallAssessment`, not at the top level.
## Errors
| Status | Body | Meaning |
| ------ | --------------------------------------------------------------------------------- | ------------------------------------- |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
| `404` | `{ "error": "No analysis found for this run. POST /runs/:runId/analyze first." }` | The run exists but was never analyzed |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/report" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
report = client.evaluations.get_report("rK7dP2qWx9TzB4mV6nJcE")
print(report.summary)
for case in report.low_scoring_cases:
print(case)
```
```json 200 OK theme={null}
{
"summary": "The agent performs well on direct how-to questions but struggles with multi-step billing queries.",
"consistencyScore": 8,
"instructionAdherence": {
"score": 9,
"analysis": "The agent generally follows the support tone guidelines.",
"deviations": ["Q3 run 2: agent gave a refund without checking eligibility"],
"rating": "high"
},
"responsePatterns": {
"similarities": ["Consistent greeting and closing"],
"differences": ["Detail level varies on billing answers"],
"outliers": ["Q3 run 2"],
"rating": "medium"
},
"reasoningAnalysis": {
"cotQuality": "Reasoning is visible and mostly sound.",
"reasoningPatterns": ["Looks up KB before answering"],
"reasoningGaps": ["Skips eligibility checks under time pressure"],
"rating": "medium"
},
"toolUsageAnalysis": {
"effectiveness": "KB search used appropriately on most cases.",
"patterns": ["search_kb before every answer"],
"issues": ["No billing tool call on refund cases"],
"rating": "medium"
},
"recommendations": [
{
"category": "instructions",
"priority": "high",
"recommendation": "Add a guardrail that requires an eligibility check before issuing refunds",
"reasoning": "One run issued a refund without confirming the customer's eligibility window"
}
],
"instructionChanges": [],
"overallAssessment": {
"strengths": ["Consistent tone", "Accurate password reset guidance"],
"weaknesses": ["Billing edge cases", "Multi-step reasoning"],
"rating": "medium"
},
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"status": "completed",
"statistics": {
"numberOfRuns": 6,
"averageRating": 7.8,
"minRating": 5,
"maxRating": 10,
"ratingVariance": 1.96
},
"lowScoringCases": [
{
"query": "Why was I charged twice?",
"response": "You may have been charged twice due to a pending authorization.",
"rating": 5,
"justification": "Agent did not escalate to the billing team as required."
}
]
}
```
```json 404 No analysis yet theme={null}
{
"error": "No analysis found for this run. POST /runs/:runId/analyze first."
}
```
# Get Run
Source: https://developers.agentx.so/api-reference/custom-eval/get-run
GET /api/v1/custom-agent-evaluations/runs/{runId}
Get a single evaluation run with statistics and per-result scores
Returns one run: its status, rating statistics, per-case repetition spread, and every submitted
result with its scores. Poll this while a run is `"in_progress"`, or read it back after
[finalizing](/api-reference/custom-eval/finalize-run).
## Authentication
Project API key.
## Path Parameters
Run ID returned by `POST /runs`.
## Response
| Field | Type | Description |
| ---------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_id` | string | Run ID |
| `runId` | string | Same value as `_id` (historical key, kept for SDK compatibility) |
| `datasetId` | string | Dataset the run scores against |
| `evaluationSettingsId` | string \| null | Standalone grading config, if one was passed to `POST /runs` |
| `scorerGroupId` | string \| null | Scorer group grading this run, if one was passed to `POST /runs` ([Scorer groups](/monitor/scorer-groups)) |
| `additionalScorerIds` | string\[] \| null | Extra judge scorer ids passed to `POST /runs`, if any |
| `scorerBreakdown` | array \| null | Per-scorer aggregate when the run used multiple judges, primary first: `[{ scorerId, name, primary, averageRating, scored }]` |
| `evaluationSubject` | object \| null | The subject metadata passed to `POST /runs` (including any `split` tag) |
| `status` | string | `"in_progress"` \| `"completed"` \| `"failed"` |
| `resultCount` | number | Total submitted results (all statuses) |
| `averageRating` | number \| null | Mean judge rating across rated results |
| `liveStatistics` | object | `{ averageRating, minRating, maxRating, ratedCount, skippedCount, failedCount }`. `skippedCount` = results the judge could not score (e.g. missing judge key or reference); `failedCount` = results submitted with an `error` |
| `caseStatistics` | array | Per-case repetition spread, for cases with 2+ rated rows (smoke-test variants excluded): `[{ questionIndex, ratedCount, averageRating, minRating, maxRating, ratingVariance }]` |
| `results` | array | Per-result rows, see below |
Each result row:
| Field | Type | Description |
| --------------------------------------------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `questionIndex` | number \| null | 0-based dataset question index |
| `questionText` | string \| null | The question's `query`, resolved from the dataset |
| `runNumber` | number \| null | Repetition number (1 through `numberOfRequests`) |
| `input` | object \| null | As submitted, e.g. `{ "query": "..." }` |
| `output` | object \| null | As submitted, e.g. `{ "text": "..." }` |
| `rating` | number \| null | Judge score 0-10; `null` when skipped/failed |
| `justification` | string \| null | Judge explanation, or the skip reason |
| `traceId` | string \| null | Linked trace, if the result carried one |
| `latencyMs` / `inputTokens` / `outputTokens` | number \| null | From the submitted `timings` |
| `vectorSimilarity` / `jaccardSimilarity` / `bleuScore` / `rougeScore` | number \| null | Similarity scores, when the metric is enabled and the case has `expectedResults` |
| `codeScorerResults` | array \| null | Code scorer, trajectory match, and context match rows: `[{ name, score, reasoning?, error? }]` |
| `judgeScorerResults` | array \| null | One verdict per additional judge scorer: `[{ scorerId, name, rating, justification, status }]`. The primary scorer's verdict stays in `rating`/`justification` |
| `isSmokeTestVariant` | boolean | Whether this row is a paraphrase variant |
| `smokeTestVariantText` | string \| null | The variant phrasing, if so |
| `status` | string | `"scored"`, `"skipped"` (judge could not score), or `"failed"` (result carried an `error`) |
| `error` | object \| null | The submitted `{ type, message }` error, if any |
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------ | ---------------------------------- |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
run = client.evaluations.get_run("rK7dP2qWx9TzB4mV6nJcE")
print(run["status"], run["averageRating"])
```
```json 200 OK theme={null}
{
"_id": "rK7dP2qWx9TzB4mV6nJcE",
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSettingsId": null,
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"framework": "langchain"
},
"status": "completed",
"resultCount": 1,
"averageRating": 8,
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
},
"caseStatistics": [],
"results": [
{
"questionIndex": 0,
"questionText": "How do I reset my password?",
"runNumber": 1,
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login screen." },
"rating": 8,
"justification": "Accurate answer, empathetic tone.",
"traceId": "mJ3vQ8pTr2LqYw6bZk9Xd",
"latencyMs": 1240,
"inputTokens": 150,
"outputTokens": 45,
"vectorSimilarity": null,
"jaccardSimilarity": 0.62,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null,
"isSmokeTestVariant": false,
"smokeTestVariantText": null,
"status": "scored",
"error": null
}
]
}
```
```json 404 Not found theme={null}
{
"error": "Run not found"
}
```
# List Datasets
Source: https://developers.agentx.so/api-reference/custom-eval/list-datasets
GET /api/v1/custom-agent-evaluations/datasets
List the project's evaluation datasets
Returns every dataset in the project selected by your API key, newest first. Use the `_id` of
the dataset you want as `datasetId` when [creating a run](/api-reference/custom-eval/create-run).
## Authentication
Project API key.
## Response
Full dataset documents (including `questions`), newest first. Each has:
| Field | Type | Description |
| --------------------------------------------------------------------- | ------ | ------------------------------------------------------------------- |
| `_id` | string | Dataset ID; pass as `datasetId` when creating runs |
| `name` | string | Display name |
| `description` | string | Description (omitted when unset) |
| `numberOfRequests` | number | Repetitions per question (default 1) |
| `acceptanceCriteria` | string | What a good response looks like (omitted when unset) |
| `rejectionCriteria` | string | What a bad response looks like (omitted when unset) |
| `evaluationCriteria` | string | Scoring rubric (omitted when unset) |
| `vectorSimilarity` / `jaccardSimilarity` / `bleuScore` / `rougeScore` | object | Present only when the metric is enabled, e.g. `{ "enabled": true }` |
| `codeScorers` | array | Configured code scorers (omitted when none) |
| `questions` | array | The dataset's question objects |
| `status` | string | Always `"published"` on self-host |
| `createdAt` | string | ISO 8601 timestamp |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/datasets" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
datasets = client.evaluations.datasets.list()
for d in datasets:
print(d.id, d.name)
```
```json 200 OK theme={null}
{
"datasets": [
{
"_id": "dS4tG7hNb2VxZ8kQ5wMyA",
"name": "Customer Support Q3 2026",
"description": "Core support questions for tier-1 agents",
"numberOfRequests": 3,
"acceptanceCriteria": "Accurate, empathetic, resolves the issue",
"rejectionCriteria": "Hallucinates, ignores the question, rude",
"jaccardSimilarity": { "enabled": true },
"questions": [
{ "main_question": { "query": "How do I reset my password?", "expectedResults": "Click Forgot Password on the login screen." } }
],
"status": "published",
"createdAt": "2026-08-01T09:00:00.000Z"
}
]
}
```
# List Models
Source: https://developers.agentx.so/api-reference/custom-eval/list-models
GET /api/v1/agent-monitoring/portability/models
List the project's model catalog used for judge selection and cost estimation
Returns the model catalog: the models selectable as judge models (for grading configs, AI
analysis, and online evaluators) and the pricing rows used for cost estimation on traces and
model-portability replays. The catalog is seeded with a small default set on first boot and is
editable from the dashboard's Settings pricing panel (or via `POST`/`PUT`/`DELETE` on this same
path).
Use a model's `id` (the exact model string sent to the provider's API) wherever a `judgeModel`
is accepted. The default judge model is the catalog's `isDefault` row.
The hosted platform's `GET /custom-agent-evaluations/models` route is **not available** on
the self-host engine and returns `404` - this catalog endpoint is the replacement. The
Python SDK's `client.evaluations.list_models()` targets the hosted route; on self-host,
read the catalog from this endpoint directly.
## Authentication
Project API key.
## Response
The default row first, alphabetical after that. Each model object has:
| Field | Type | Description |
| --------------------------- | -------------- | ------------------------------------------------------------------------------ |
| `id` | string | Exact model string sent to the provider's API - use this as `judgeModel` |
| `provider` | string | `"openai"`, `"anthropic"`, `"gemini"`, or `"custom"` |
| `label` | string | Human-readable name |
| `pricePerMInputTokens` | number | USD per 1M input tokens |
| `pricePerMOutputTokens` | number | USD per 1M output tokens |
| `pricePerMCacheReadTokens` | number \| null | USD per 1M cache-read tokens; `null` falls back to the input rate |
| `pricePerMCacheWriteTokens` | number \| null | USD per 1M cache-write tokens; `null` falls back to the input rate |
| `isDefault` | boolean | Whether this is the default judge model |
| `baseUrl` | string \| null | Custom OpenAI-compatible endpoint, for `"custom"` models |
| `apiKeyMasked` | string \| null | Masked per-model API key, if one is configured (the raw key is never returned) |
## Related
* `GET /api/v1/agent-monitoring/portability/models/unpriced` - models seen on token-bearing
traces in the last 30 days that have no catalog pricing (so unpriced spend is visible
instead of a silent \$0).
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/agent-monitoring/portability/models" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```json 200 OK theme={null}
{
"models": [
{
"id": "gpt-5.6-luna",
"provider": "openai",
"label": "GPT-5.6 Luna",
"pricePerMInputTokens": 1.25,
"pricePerMOutputTokens": 10.0,
"pricePerMCacheReadTokens": null,
"pricePerMCacheWriteTokens": null,
"isDefault": true,
"baseUrl": null,
"apiKeyMasked": null
},
{
"id": "claude-opus-4-7",
"provider": "anthropic",
"label": "Claude Opus 4.7",
"pricePerMInputTokens": 15.0,
"pricePerMOutputTokens": 75.0,
"pricePerMCacheReadTokens": 1.5,
"pricePerMCacheWriteTokens": 18.75,
"isDefault": false,
"baseUrl": null,
"apiKeyMasked": null
}
]
}
```
# List Runs
Source: https://developers.agentx.so/api-reference/custom-eval/list-runs
GET /api/v1/custom-agent-evaluations/runs
List the project's evaluation runs, newest first
Returns the project's most recent evaluation runs (up to 50), newest first. Each entry is the
same full run summary [Get Run](/api-reference/custom-eval/get-run) returns, including
`liveStatistics` and per-result rows.
There is no pagination on this endpoint - it returns at most the 50 newest runs, and query
parameters are ignored. Fetch a specific run by id with
[Get Run](/api-reference/custom-eval/get-run).
## Authentication
Project API key.
## Response
Up to 50 run objects, newest first, each in the
[Get Run](/api-reference/custom-eval/get-run) shape: `_id`/`runId`, `datasetId`,
`evaluationSettingsId`, `evaluationSubject`, `status`
(`"in_progress"` | `"completed"` | `"failed"`), `resultCount`, `averageRating`,
`liveStatistics`, `caseStatistics`, and `results`.
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```json 200 OK theme={null}
{
"runs": [
{
"_id": "rK7dP2qWx9TzB4mV6nJcE",
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
"evaluationSettingsId": null,
"evaluationSubject": {
"kind": "custom_agent",
"displayName": "Customer Support Bot",
"framework": "langchain"
},
"status": "completed",
"resultCount": 6,
"averageRating": 7.8,
"liveStatistics": {
"averageRating": 7.8,
"minRating": 5,
"maxRating": 10,
"ratedCount": 6,
"skippedCount": 0,
"failedCount": 0
},
"caseStatistics": [],
"results": [ { "questionIndex": 0, "rating": 8, "...": "..." } ]
}
]
}
```
# Get Missing Results
Source: https://developers.agentx.so/api-reference/custom-eval/missing-results
GET /api/v1/custom-agent-evaluations/runs/{runId}/missing-results
Resume support: the idempotency keys a run has already accepted
Returns the idempotency keys this run has already stored, so a client resuming after a crash or
network failure can skip re-running (and re-paying for) already-submitted cases. The Python
SDK's `execute()` derives its keys deterministically as `{runId}:{caseId}:run-{runNumber}` and
uses exactly this endpoint to resume.
The engine cannot know your full case list, so it reports what was **submitted**, not what is
missing: the legacy `missing` field is always an empty array. Compute missing cases
client-side as your own expected keys minus `submittedKeys`.
## Authentication
Project API key.
## Path Parameters
Run ID.
## Response
Echoed back.
The `idempotencyKey` of every result this run has accepted.
`submittedKeys.length`.
Always `[]`. Legacy field kept for older SDK versions; the client computes missing cases
from `submittedKeys`.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------ | ---------------------------------- |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/missing-results" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Resume pattern (Python SDK) theme={null}
from agentx.evaluations.models import EvaluationResult
submitted = set(client.evaluations.get_submitted_keys(run_id))
for i, question in enumerate(questions):
key = f"{run_id}:case-{i}:run-1"
if key in submitted:
continue # already scored - skip re-running and re-paying
output = my_agent(question)
client.evaluations.append_results(run_id, batch_id=f"resume-{run_id}", results=[
EvaluationResult(
case_id=f"case-{i}",
question_index=i,
run_number=1,
idempotency_key=key,
input={"query": question},
output={"text": output},
)
])
```
```json 200 OK theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"submittedKeys": [
"rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rK7dP2qWx9TzB4mV6nJcE:case-1:run-1"
],
"submittedCount": 2,
"missing": []
}
```
```json 404 Not found theme={null}
{
"error": "Run not found"
}
```
# Custom Evaluations Overview
Source: https://developers.agentx.so/api-reference/custom-eval/overview
Run evaluations programmatically against any external agent and score with the LLM judge
The Custom Agent Evaluation API runs evaluations against **any agent you own**, regardless of
framework: your code drives the agent, submits its outputs, and the engine scores each result
synchronously with the LLM judge (plus optional similarity metrics and code scorers). A
detailed AI analysis report can be generated after the run, and a
[CI gate](/api-reference/ci-cd/get-run) can turn the finished run into a pass/fail verdict.
Examples target a local self-host engine (`http://localhost:4700`). Substitute your own
deployment's base URL. The Python SDK wraps this entire surface as `client.evaluations` -
see the [Evaluations SDK](/sdk/evaluations/overview).
## When to use this API
| Goal | Use |
| ------------------------------------------------------------- | ----------------------------------------------------------------------- |
| Run a multi-question evaluation and get a score report | This API (or the [Evaluations SDK](/sdk/evaluations/overview)) |
| Gate a PR or deploy on eval quality | The same run, then the [CI gate endpoint](/api-reference/ci-cd/get-run) |
| Score one already-ingested trace without re-running the agent | [Evaluate Trace](/api-reference/tracing/evaluate-trace) |
| Record traces for browsing | [Submit Trace](/api-reference/tracing/submit-trace) |
## Base path
```
/api/v1/custom-agent-evaluations
```
## Authentication
All routes require a project API key in the `x-api-key` header:
```http theme={null}
x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8
```
## Lifecycle
```
POST /datasets ← create (or reuse) a dataset with your questions
POST /evaluation-settings ← optional: a reusable grading config, independent of any dataset
│
POST /runs ← start a run
│ (pass evaluationSettingsId to grade with a config above
│ instead of the dataset's own criteria)
POST /runs/:id/results ← submit batches of agent outputs; each result is judge-scored
│ synchronously and every batch returns liveStatistics
POST /runs/:id/finalize ← mark the run complete (idempotent; also returns liveStatistics)
│
POST /runs/:id/analyze ← run AI analysis (synchronous on self-host)
GET /runs/:id/report ← fetch the full report: narrative analysis + statistics
GET /runs/:id/gate ← optional: pass/fail verdict for CI
```
## Limits
| Limit | Value |
| ---------------------------------- | --------------------------------------------------- |
| Max batch size per `/results` call | 10 results |
| AI analysis sample | worst 12 + best 5 rated results in the judge prompt |
| Judge models per analysis | up to 3 |
The judge scores each result synchronously in the `/results` call, so batches of judged results
take as long as the underlying LLM calls. Daily judge spend can be capped with
`AGENTX_QUOTA_JUDGE_CALLS_PER_DAY` (see [Authentication](/authentication)).
## Endpoints
| Method | Path | Description |
| -------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `GET` | `/datasets` | [List datasets](/api-reference/custom-eval/list-datasets) |
| `POST` | `/datasets` | [Create dataset](/api-reference/custom-eval/create-dataset) |
| `GET` | `/datasets/:id` | [Get dataset](/api-reference/custom-eval/get-dataset) |
| `DELETE` | `/datasets/:id` | Delete a dataset (`409` if its grading config is attached to a live scorer) |
| `POST` | `/datasets/:id/cases` | Append a curated case to a dataset (with dedupe) |
| `GET` | `/evaluation-settings` | List standalone grading configs |
| `POST` | `/evaluation-settings` | Create a standalone grading config (criteria, judge prompt/model, similarity metrics, code scorers), reusable across datasets |
| `GET` | `/evaluation-settings/:id` | Get a grading config |
| `POST` | `/runs` | [Create run](/api-reference/custom-eval/create-run) |
| `GET` | `/runs` | [List runs](/api-reference/custom-eval/list-runs) |
| `GET` | `/runs/:id` | [Get run](/api-reference/custom-eval/get-run) |
| `POST` | `/runs/:id/results` | [Submit results](/api-reference/custom-eval/submit-results) |
| `GET` | `/runs/:id/missing-results` | [Resume support: submitted idempotency keys](/api-reference/custom-eval/missing-results) |
| `POST` | `/runs/:id/finalize` | [Finalize run](/api-reference/custom-eval/finalize-run) |
| `POST` | `/runs/:id/analyze` | [Analyze run](/api-reference/custom-eval/analyze-run) |
| `GET` | `/runs/:id/analyze-status` | Analysis status (terminal immediately on self-host; see [Analyze Run](/api-reference/custom-eval/analyze-run)) |
| `GET` | `/runs/:id/report` | [Get report](/api-reference/custom-eval/get-report) |
| `GET` | `/runs/:id/gate` | [CI gate: pass/fail the run](/api-reference/ci-cd/get-run) |
| `POST` | `/prompts` | Create a prompt in the prompt registry |
| `GET` | `/prompts` | List prompts |
| `GET` | `/prompts/:identifier` | Get a prompt by name or id (`?version=` for a specific version) |
The judge model catalog lives on a separate route - see
[List Models](/api-reference/custom-eval/list-models).
# Submit Results
Source: https://developers.agentx.so/api-reference/custom-eval/submit-results
POST /api/v1/custom-agent-evaluations/runs/{runId}/results
Submit a batch of agent outputs for synchronous judge scoring
Submits up to 10 agent outputs for a run. Each new result is scored **synchronously inside this
request** - a judge LLM call per result, plus any enabled similarity metrics and code scorers -
so allow a generous client timeout for judged batches. Submissions are idempotent: resubmitting
an `idempotencyKey` the run has already accepted returns the stored score without re-scoring.
Send results as you collect them rather than accumulating everything client-side: every batch
response carries `liveStatistics`, so the run's average rating is observable while the run is
still executing.
## Authentication
Project API key.
## Path Parameters
Run ID returned by `POST /runs`.
## Body
Identifier for this batch, stored on each result row.
Array of 1-10 result objects. Each result has:
Unique key for this result within the run. Prevents duplicate scoring on retry (the
engine also enforces uniqueness on `(runId, idempotencyKey)` at insert, so a racing
duplicate counts as a duplicate, never an error). Recommended format:
`{runId}:{caseId}:run-{runNumber}`. A result without one is counted in
`failedValidation` and dropped.
0-based index into the dataset's `questions` array. Selects the case's
`expectedResults`, `judgeGuideline`, and other per-case grading inputs.
Which repetition this is (1 through `numberOfRequests`).
Case identifier. Defaults to `case-{questionIndex}`.
Agent input: `{ "query": string }`.
Agent output: `{ "text": string }`. Required unless `error` is set - a result with
neither is counted in `failedValidation` and dropped.
Error that occurred: `{ "type": string, "message": string }`. When set, the result is
stored with status `"failed"` and rating 0; no judge call is made.
ID of a trace previously ingested via
[Submit Trace](/api-reference/tracing/submit-trace) (the Python SDK's
`client.tracer.trace(..., sync=True)` returns it as `span.trace_id`). Linking a trace
unlocks trajectory-aware judging (the judge sees the actual tool-call path), the
trajectory and context match scorers, tool definitions for `toolContext: "detailed"`
configs, and the dashboard's View Trace on the result.
What the agent actually retrieved for this case - a string or a list of chunk strings.
Feeds `{context}`-referencing judge prompts (the RAG metric pack) and the context match
scorer. When absent, the engine falls back to the linked trace's recorded retrieval
spans, then the case's pinned `retrievalContext`.
Performance metrics: `{ "latencyMs": number, "inputTokens": number, "outputTokens": number }`.
Marks this row as a paraphrase variant from the run's `smokeTestVariants`. Variant rows
are excluded from case statistics, low-scoring cases, and CI-gate aggregates.
The variant phrasing that was actually asked, when `isSmokeTestVariant` is `true`.
The hosted platform's `observableTrace` and `metadata` result fields are ignored by the
self-host engine - they are accepted without error but not stored. Link a real trace via
`traceId` instead; it powers strictly more (trajectory judging, timeline, scorers).
## Response
Echoed back.
Echoed back.
New results scored and stored in this call.
Results skipped because their `idempotencyKey` already existed on this run.
Results dropped (missing `idempotencyKey`, or both `output.text` and `error` absent).
The run's status, e.g. `"in_progress"`.
Per-result scores, in submission order (validation-dropped results are omitted):
`[{ idempotencyKey, rating, justification, status, vectorSimilarity, jaccardSimilarity, bleuScore, rougeScore, codeScorerResults, deduped? }]`.
`status` is `"scored"`, `"skipped"` (judge could not score - the reason is in
`justification`), or `"failed"` (the result carried an `error`). Duplicates echo the stored
first-submission payload with `deduped: true`.
Rating aggregate recomputed after this batch:
`{ averageRating, minRating, maxRating, ratedCount, skippedCount, failedCount }`. Available
on every batch response, no [Analyze Run](/api-reference/custom-eval/analyze-run) required;
the Python SDK exposes the same numbers as `run.average_rating` and friends.
## Errors
| Status | Body | Meaning |
| ------ | --------------------------------------------------- | -------------------------------------------------------------------- |
| `400` | `{ "error": "batchId is required" }` | Missing `batchId` |
| `400` | `{ "error": "results must be a non-empty array" }` | Missing or empty `results` |
| `400` | `{ "error": "Batch size must not exceed 10" }` | More than 10 results in one call |
| `404` | `{ "error": "Run not found" }` | No run with this id in the project |
| `409` | `{ "error": "Run is already in a terminal state" }` | The run is `"completed"` or `"failed"`; no more results are accepted |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE/results \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"batchId": "batch-001",
"results": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"questionIndex": 0,
"runNumber": 1,
"caseId": "case-0",
"input": { "query": "How do I reset my password?" },
"output": { "text": "Click Forgot Password on the login screen." },
"timings": { "latencyMs": 1240, "inputTokens": 150, "outputTokens": 45 },
"traceId": "mJ3vQ8pTr2LqYw6bZk9Xd"
}
]
}'
```
```json 200 OK theme={null}
{
"runId": "rK7dP2qWx9TzB4mV6nJcE",
"batchId": "batch-001",
"accepted": 1,
"duplicates": 0,
"failedValidation": 0,
"status": "in_progress",
"scoredResults": [
{
"idempotencyKey": "rK7dP2qWx9TzB4mV6nJcE:case-0:run-1",
"rating": 8,
"justification": "The agent correctly identified the password reset flow and gave clear step-by-step instructions.",
"status": "scored",
"vectorSimilarity": null,
"jaccardSimilarity": 0.62,
"bleuScore": null,
"rougeScore": null,
"codeScorerResults": null
}
],
"liveStatistics": {
"averageRating": 8,
"minRating": 8,
"maxRating": 8,
"ratedCount": 1,
"skippedCount": 0,
"failedCount": 0
}
}
```
```json 409 Terminal state theme={null}
{
"error": "Run is already in a terminal state"
}
```
# Evaluate Trace
Source: https://developers.agentx.so/api-reference/tracing/evaluate-trace
POST /api/v1/ingest/traces/{traceId}/evaluate
Score a recorded trace against a dataset or grading config without re-running the agent
Grades the trace's recorded input/output against a dataset's (or standalone grading config's)
criteria, as a real one-result evaluation run. The agent is **not** called again. The run is
stored with `runSource: "trace-eval"` and shows up in Evaluate > Runs like any other, but it is
never used as a no-regression baseline for full runs.
Scoring uses the full run-scoring stack: the LLM judge (which needs a judge provider key
configured), the trace's recorded tool trajectory, retrieval context, and any code scorers on
the config.
Need a `traceId`? [Submit Trace](/api-reference/tracing/submit-trace) returns one on every
call. In the Python SDK, only `client.tracer.trace(..., sync=True)` (context manager) gets it
back synchronously; the decorator form is fire-and-forget and never returns an id.
## Authentication
Project API key.
## Path Parameters
ID of the trace to evaluate, returned by `POST /ingest/traces`.
## Body
ID of the dataset **or** standalone evaluation-settings config to score against. Datasets
grade with their own criteria; a grading config brings its own judge prompt/model.
The Python SDK's `question_index` argument is sent on the wire but **ignored** by the
self-host engine - the trace is scored against the config's general criteria, not one
question's `expectedResults`.
## Response
This response keeps its historical snake\_case keys (`run_id`, `trace_id`) - they are the
actual wire keys, matching what the Python SDK's `evaluate_trace()` reads.
ID of the created one-result evaluation run.
Echoed back for confirmation.
Score from 0 (poor) to 10 (excellent). `null` when the judge could not score (for example,
no judge LLM key is configured) - the run's result row records the reason.
LLM-generated explanation of the score.
Always `"completed"` on success.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------------------------------ | ---------------------------------------------------------- |
| `400` | `{ "error": "traceId and datasetId are required" }` | Missing path or body parameter |
| `404` | `{ "error": "Trace not found" }` | No such trace in this project |
| `404` | `{ "error": "Dataset or evaluator config not found" }` | `datasetId` matches neither a dataset nor a grading config |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/ingest/traces/mJ3vQ8pTr2LqYw6bZk9Xd/evaluate \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA" }'
```
```python Python SDK theme={null}
result = client.tracer.evaluate_trace(
trace_id="mJ3vQ8pTr2LqYw6bZk9Xd",
dataset_id="dS4tG7hNb2VxZ8kQ5wMyA",
)
print(result["rating"])
print(result["justification"])
```
```json 200 OK theme={null}
{
"run_id": "rK7dP2qWx9TzB4mV6nJcE",
"trace_id": "mJ3vQ8pTr2LqYw6bZk9Xd",
"rating": 8,
"justification": "The agent correctly described the password reset flow and provided accurate step-by-step instructions.",
"status": "completed"
}
```
```json 404 Not found theme={null}
{
"error": "Trace not found"
}
```
# List Traces
Source: https://developers.agentx.so/api-reference/tracing/list-traces
GET /api/v1/ingest/traces
Cursor-paginated list of ingested traces with database-side search and filters
Returns the project's root traces, newest first, with cursor-based pagination. This is what the
dashboard's Live Traces feed (Governance > Observe) reads; use it to browse or export what your
agents ingested via [Submit Trace](/api-reference/tracing/submit-trace) or OpenTelemetry.
## Authentication
Project API key.
## Query Parameters
Page size, 1 to 100.
Trace `_id` to continue after, taken from `nextCursor` in the previous response.
Filter by platform label. Folded like the stored value (trimmed, lowercased), so
`?framework=LangChain` matches traces stored as `"langchain"`.
Database-side text search over the trace list.
`"production"` (excludes eval-run traffic), `"eval"` (eval-run traffic only), or `"all"`.
The dashboard's Live Traces sends `"production"` by default.
## Response
Total root traces matching the filters across all pages - what the dashboard's pagination
range ("1-50 of N") is computed from.
Array of trace objects, newest first. Each has:
| Field | Type | Description |
| --------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `_id` | string | Trace ID |
| `name` | string | Agent or operation label |
| `input` | any | Agent input (omitted when absent) |
| `output` | any | Agent output (omitted when absent) |
| `latencyMs` | number | Latency in ms |
| `error` | string | Error message if the run failed |
| `framework` | string | Platform label, lowercased |
| `model` | string | LLM model |
| `toolCalls` | array | Tool calls made during the run |
| `sessionId` | string | Session grouping key |
| `spanId` | string | Client-supplied span identity, if any |
| `spanKind` | string | Always present: `"agent"`, `"llm"`, `"tool"`, `"retrieval"`, `"chain"`, `"embedding"`, `"reranker"`, `"guardrail"`, `"evaluator"`, `"prompt"`, or `"memory"` |
| `judgeScores` | object \| null | Always present, nullable: the trace's live judge/scorer-group verdict summary for the Score chip - `{ rating, threshold, scorerName, judgeCount, failingCount, verdicts }`, or `null` when nothing scored it |
| `parentSpanId` | string | Parent span's id, for span-tree traces |
| `startedAt` | string | ISO 8601 span start, when the producer sent one |
| `trafficSource` | string | `"eval-run"` for traces produced inside an offline evaluation; absent for production |
| `source` | string | Always `"sdk"` on self-host |
| `createdAt` | string | ISO 8601 timestamp (the traffic's own time for historical imports) |
| `inputTokens` | number \| null | Input token count |
| `outputTokens` | number \| null | Output token count |
Optional fields are omitted, not `null` (except the two token counts and `judgeScores`, which are always
present and nullable).
Whether more traces exist after this page.
Pass as `cursor` to fetch the next page. `null` on the last page.
## Related endpoints
* `GET /api/v1/ingest/traces/{traceId}` - full single-trace detail, adding `metadata`,
`performanceSummary`, `cacheReadTokens`/`cacheWriteTokens`, `estimatedCostUSD` (null when
the model has no catalog pricing or the trace has no token counts), and `topic`
(intent/sentiment/issue classification, when topics ran). `404` if the trace doesn't exist.
* `GET /api/v1/ingest/sessions/{sessionId}/spans` - every span in one session/OTel trace as
`{ "spans": [...] }`, ordered by span start, for assembling a span tree without pagination.
## Pagination example
```http theme={null}
GET /api/v1/ingest/traces?limit=20&source=production
→ { "traces": [...], "totalCount": 143, "hasNextPage": true, "nextCursor": "mJ3vQ8pTr2LqYw6bZk9Xd" }
GET /api/v1/ingest/traces?limit=20&source=production&cursor=mJ3vQ8pTr2LqYw6bZk9Xd
→ { "traces": [...], "totalCount": 143, "hasNextPage": false, "nextCursor": null }
```
```bash cURL theme={null}
curl "http://localhost:4700/api/v1/ingest/traces?limit=20&source=production&framework=langchain" \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```json 200 OK theme={null}
{
"totalCount": 143,
"traces": [
{
"_id": "mJ3vQ8pTr2LqYw6bZk9Xd",
"name": "customer-support-agent",
"framework": "langchain",
"model": "gpt-4o",
"input": "How do I reset my password?",
"output": "Click Forgot Password on the login page.",
"latencyMs": 1340,
"toolCalls": [
{ "name": "search_kb", "input": "password reset", "output": "...", "success": true }
],
"sessionId": "user_session_abc123",
"spanKind": "agent",
"judgeScores": null,
"source": "sdk",
"createdAt": "2026-08-27T10:30:00.000Z",
"inputTokens": 150,
"outputTokens": 45
}
],
"hasNextPage": true,
"nextCursor": "mJ3vQ8pTr2LqYw6bZk9Xd"
}
```
# Submit Trace
Source: https://developers.agentx.so/api-reference/tracing/submit-trace
POST /api/v1/ingest/traces
Ingest a single span or agent run into the engine's trace store
Records one span (an agent run, LLM call, tool call, or retrieval step) in the project selected
by your API key. This is the endpoint the Python SDK's `client.tracer.trace(...)` posts to, and
the one to call directly from any language the SDK doesn't cover. The response returns the
stored trace's id, which you can pass to
[Evaluate Trace](/api-reference/tracing/evaluate-trace) or link to an evaluation result.
**Casing**: camelCase keys (`latencyMs`, `sessionId`, ...) are the canonical wire form, and
the snake\_case spellings (`latency_ms`, `session_id`, ...) are accepted as legacy aliases on
this endpoint only, for existing SDKs. If a request carries both spellings of a key, the
snake\_case value wins. New integrations should send camelCase.
## Authentication
Project API key. The key alone selects the project; no project id is sent.
## Body
Agent or operation label. For a root span, the name resolves to a registered agent
(auto-registering one on first use) unless `agentId` says otherwise.
Agent input: string, JSON object, array, etc.
Agent output.
Error message if the run failed.
End-to-end wall-clock latency in milliseconds. Legacy alias: `latency_ms`.
Platform label - an SDK integration literal (`"langchain"`, `"crewai"`, `"openai-agents"`,
`"anthropic"`, ...), an OTel scope name, or any custom string for platforms AgentX has no
integration for. Trimmed, folded to lowercase, and capped at 64 characters on ingest, so
`"LangChain"` and `"langchain"` chart and filter as one platform. Unlabeled traces bucket as
"other" in the dashboard. See [Platform detection](/trace/platform-detection).
LLM model used, e.g. `"gpt-4o"`. Must match a pricing-catalog model id for cost estimation.
Tool calls made during the run, as an array of objects (any keys; `name`, `input`, `output`,
and `success` are what the monitor and judges read). Legacy alias: `tool_calls`.
What kind of step this span is: `"agent"`, `"llm"`, `"tool"`, `"retrieval"`, `"chain"`,
`"embedding"`, `"reranker"`, `"guardrail"`, `"evaluator"`, `"prompt"`, or `"memory"`. Other vocabularies
(OpenInference, OTel GenAI, Langfuse, MLflow) are folded onto these on ingest; unrecognized
values are stored as null. Legacy alias: `span_kind`. See
[Span kinds](/trace/span-kinds).
Where the trace came from. `"eval-run"` marks traffic produced inside an offline evaluation
run; monitoring and the Live Traces production filter exclude it. Unknown values are stored
as null.
Groups spans from the same user session, conversation thread, or OTel trace. Legacy alias:
`session_id`.
Arbitrary key-value metadata, stored on the trace and passed to online evaluators.
Arbitrary performance breakdown object, stored verbatim. Legacy alias:
`performance_summary`.
Input token count. Legacy alias: `input_tokens`.
Output token count. Legacy alias: `output_tokens`.
Prompt-cache read tokens - a subset of `inputTokens`, not additional tokens. Priced at the
model's cache-read rate when one is configured. Legacy alias: `cache_read_tokens`.
Prompt-cache write tokens - also a subset of `inputTokens`. Legacy alias:
`cache_write_tokens`.
Stable client-supplied span identity. Makes ingest idempotent: replaying the same `spanId`
stores nothing new, triggers no duplicate monitoring or judging, and returns the original
trace id with `deduped: true`. Legacy alias: `span_id`.
The parent span's `spanId`, for a real span hierarchy (OTel ingestion sends this
unconditionally; the Python SDK sends it with `span_tree=True`). Child spans skip agent
resolution, the daily trace quota, and the monitoring pipeline. Legacy alias:
`parent_span_id`.
Span start time as a Unix-epoch nanosecond count, sent as a **string** (the value exceeds
safe-integer precision as a JSON number). When present it also becomes the trace's
`createdAt`, so historical imports land in the right time window. Legacy alias:
`started_at_unix_nano`.
Explicit agent to attribute this root span to - a registered agent id from
[Register Agent](/api-reference/tracked-agents/add), or an agent name. Omit to resolve from
`name`, identical to the pre-registry behavior. Legacy alias: `agent_id`.
`false` skips every ingest-time check (pattern detection, online and custom evaluators,
topics) for this trace - eval harnesses send this. `true` opts eval-run traffic back into
monitoring, and combined with `patternIds` restricts detection to those patterns. Omitted:
the default sweep runs on every root span.
Custom pattern ids to restrict detection to. Only meaningful alongside `monitor: true`.
Legacy alias: `pattern_ids`.
Accepted for wire compatibility with the hosted SaaS payload shape; ignored by the self-host
engine (the API key already selects the project).
**Payload caps**: `input`, `output`, `toolCalls`, and `metadata` are each capped at 100,000
characters serialized (`AGENTX_INGEST_MAX_FIELD_CHARS`). Oversized fields are stored
truncated with an explicit `agentx.truncated` marker, never rejected.
## Response
Returns `200 OK` once the span is durably accepted (monitoring and evaluator checks run in the
background after the response).
ID of the stored trace. Canonical key.
The same id under its legacy snake\_case key, kept for existing SDKs. Both keys are always
present and always equal.
`true` when a span with the same `spanId` was already stored; the returned id is the
original trace's, and no monitoring or judging re-ran.
## Errors
| Status | Body | Meaning |
| ------ | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| `422` | `{ "error": "Invalid trace payload", "details": ... }` | Schema validation failed (e.g. missing `name`) |
| `429` | `{ "error": "Ingest queue is full - retry with backoff (Retry-After: 1s)." }` | Backpressure; retry after the `Retry-After` header (1s). Nothing was stored |
| `429` | `{ "error": "Daily trace quota reached (...)" }` | `AGENTX_QUOTA_TRACES_PER_DAY` hit; resets at midnight. Child spans are exempt |
| `503` | `{ "error": "Trace storage is unavailable - the span was not stored; retry." }` | Storage flush failed; retry after the `Retry-After` header (2s) |
Retries are safe: send a `spanId` and a redelivered span dedupes instead of duplicating.
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/ingest/traces \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{
"name": "customer-support-agent",
"framework": "langchain",
"model": "gpt-4o",
"input": "How do I reset my password?",
"output": "Click Forgot Password on the login page.",
"latencyMs": 1340,
"inputTokens": 150,
"outputTokens": 45,
"toolCalls": [
{ "name": "search_kb", "input": "password reset", "output": "...", "success": true }
],
"sessionId": "user_session_abc123"
}'
```
```python Python SDK (decorator) theme={null}
from agentx import AgentX
client = AgentX.from_env()
@client.tracer.trace("customer-support-agent", framework="langchain", model="gpt-4o")
def handle(query: str) -> str:
return chain.invoke(query)
handle("How do I reset my password?")
```
```python Python SDK (get trace_id back) theme={null}
from agentx import AgentX
client = AgentX.from_env()
# sync=True blocks until the trace is ingested, so trace_id is available as soon as the
# `with` block exits. The decorator form above is fire-and-forget and never returns an id.
with client.tracer.trace(
"customer-support-agent", framework="langchain", model="gpt-4o", sync=True
) as span:
span.output = chain.invoke("How do I reset my password?")
print(span.trace_id) # "mJ3vQ8pTr2LqYw6bZk9Xd"
```
```json 200 OK theme={null}
{
"trace_id": "mJ3vQ8pTr2LqYw6bZk9Xd",
"traceId": "mJ3vQ8pTr2LqYw6bZk9Xd",
"deduped": false
}
```
```json 422 Invalid payload theme={null}
{
"error": "Invalid trace payload",
"details": {
"fieldErrors": { "name": ["Required"] },
"formErrors": []
}
}
```
```json 429 Queue full theme={null}
{
"error": "Ingest queue is full - retry with backoff (Retry-After: 1s)."
}
```
```json 503 Storage unavailable theme={null}
{
"error": "Trace storage is unavailable - the span was not stored; retry."
}
```
# Register Agent
Source: https://developers.agentx.so/api-reference/tracked-agents/add
POST /api/v1/agents
Explicitly register an agent identity in the project
Creates a new agent row and returns its id. Registration is optional: tracing under a bare
`name` auto-registers a single stable agent per distinct name. Register explicitly when you
want a real id up front, or when you deliberately need **two agents sharing one display name**
(the only way that can happen) - from then on, disambiguate them by passing the returned `_id`
as `agentId` on [Submit Trace](/api-reference/tracing/submit-trace).
This endpoint always creates a **new** row, even if an agent with this name already exists.
The implicit path (tracing under a name with no explicit `agentId`) keeps resolving to the
oldest agent registered under that name, so existing callers never change behavior.
## Authentication
Project API key.
## Body
Agent display name. Leading/trailing whitespace is trimmed.
## Response
Returns `201 Created`.
The created agent: `{ "_id": string, "name": string, "createdAt": string }`.
## Errors
| Status | Body | Meaning |
| ------ | --------------------------------- | -------------------------------------- |
| `400` | `{ "error": "name is required" }` | `name` missing, not a string, or blank |
```bash cURL theme={null}
curl -X POST http://localhost:4700/api/v1/agents \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8" \
-H "Content-Type: application/json" \
-d '{ "name": "customer-support-agent" }'
```
```python Python SDK theme={null}
agent = client.monitor.create_agent("customer-support-agent")
print(agent["_id"])
```
```json 201 Created theme={null}
{
"agent": {
"_id": "aG5kR8sLp3WvX1nC7qZtB",
"name": "customer-support-agent",
"createdAt": "2026-08-27T10:30:00.000Z"
}
}
```
```json 400 Missing name theme={null}
{
"error": "name is required"
}
```
# List Agents
Source: https://developers.agentx.so/api-reference/tracked-agents/list
GET /api/v1/agents
List every agent the project knows about, with each one's monitoring profile
Returns the project's agent registry, sorted by name. An agent row exists for every distinct
agent the engine has seen: agents are **auto-registered** the first time a root trace arrives
under a new name, and can also be registered explicitly with
[Register Agent](/api-reference/tracked-agents/add). Use this list to find the agent ids that
monitoring profiles, patterns, and online-evaluator scopes key off.
There is no separate "tracked agents" list on the self-host engine - every registered agent's
traces appear in Live Traces. This endpoint replaces the hosted platform's
`/ingest/tracked-agents` surface.
## Authentication
Project API key.
## Response
Array of agent objects, sorted by name. Each has:
| Field | Type | Description |
| ------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `_id` | string | Agent ID - pass as `agentId` on trace ingest, monitoring profiles, and scope arrays |
| `name` | string | Display name (two agents may deliberately share one; see [Register Agent](/api-reference/tracked-agents/add)) |
| `kind` | string | Always `"agent"` |
| `agentType` | string | Always `"external"` on self-host (agents arrive via SDK/dashboard registration, never a native agent builder) |
| `monitoringAgentId` | string | Same value as `_id` |
| `monitoringProfile` | object \| null | The agent's monitoring profile, or `null` when none has been configured. Includes `enabled`, `failureDetectionEnabled`, `infoDetectionEnabled`, `topicsEnabled`, `sampleRate`, `retentionDays`, `thresholdOverrides`, `channels`, and timestamps |
| `createdAt` | string | ISO 8601 timestamp |
```bash cURL theme={null}
curl http://localhost:4700/api/v1/agents \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```python Python SDK theme={null}
agents = client.monitor.list_agents()
for agent in agents:
print(agent["_id"], agent["name"])
```
```json 200 OK theme={null}
{
"agents": [
{
"_id": "aG5kR8sLp3WvX1nC7qZtB",
"name": "customer-support-agent",
"kind": "agent",
"agentType": "external",
"monitoringAgentId": "aG5kR8sLp3WvX1nC7qZtB",
"monitoringProfile": null,
"createdAt": "2026-08-27T10:30:00.000Z"
}
]
}
```
# Get Agent
Source: https://developers.agentx.so/api-reference/tracked-agents/remove
GET /api/v1/agents/{id}
Fetch a single registered agent by id
Returns one agent from the project's registry. Use it to resolve an id you stored earlier (for
example, from [Register Agent](/api-reference/tracked-agents/add) or a monitoring signal's
`agentId`) back to its display name.
Agents cannot be deleted through the API - the registry row is the identity every trace,
monitoring profile, and signal for that agent keys off. The hosted platform's
`DELETE /ingest/tracked-agents/:botId` endpoint does not exist on the self-host engine; to
quiet an agent, disable its monitoring profile instead
(`PUT /api/v1/monitor/profiles/{agentId}` with `{ "enabled": false }`).
## Authentication
Project API key.
## Path Parameters
Agent `_id` from [List Agents](/api-reference/tracked-agents/list).
## Response
The agent: `{ "_id": string, "name": string, "createdAt": string }`.
## Errors
| Status | Body | Meaning |
| ------ | -------------------------------- | ------------------------------------ |
| `404` | `{ "error": "Agent not found" }` | No agent with this id in the project |
```bash cURL theme={null}
curl http://localhost:4700/api/v1/agents/aG5kR8sLp3WvX1nC7qZtB \
-H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"
```
```json 200 OK theme={null}
{
"agent": {
"_id": "aG5kR8sLp3WvX1nC7qZtB",
"name": "customer-support-agent",
"createdAt": "2026-08-27T10:30:00.000Z"
}
}
```
```json 404 Not found theme={null}
{
"error": "Agent not found"
}
```
# Authentication
Source: https://developers.agentx.so/authentication
Project API keys, the dashboard connect flow, and multi-user mode (AGENTX_AUTH)
Everything in AgentX authenticates with a **project API key** sent as an `x-api-key` header.
The dashboard, the Python SDK, CI pipelines, and OpenTelemetry exporters all use the same key.
```http theme={null}
x-api-key: agtx_local_...
```
## Where keys come from
* **Engine startup log**: the line `Default project API key: agtx_local_...` prints on every
boot. This is the canonical copy source for SDK and CI callers on a fresh instance.
* **`GET /api/v1/auth/config`**: in the default (no-auth) mode, this unauthenticated endpoint
includes the default project's `apiKey` - a deliberate tradeoff: anyone who can reach the
port already owns the instance in practice, so bind it accordingly (see
[Security model](#security-model)). In `AGENTX_AUTH=enabled` mode, no key is ever handed
out this way.
* **Dashboard**: once connected, **Settings → API access** shows the current project's key,
and each project you create gets its own key (the project switcher moves between them).
## Dashboard: connecting
The default self-host mode has no user accounts. On first visit the dashboard reads
`/api/v1/auth/config`, receives the default project's key, and connects automatically - a
fresh install lands on a working screen with zero setup. The key is kept in that browser's
local storage and attached to every request; the sidebar's **Disconnect** button forgets it.
A manual connect screen (paste a key from the startup log) remains as a fallback for older
engines and for connecting with a specific project's key.
## SDK
```python theme={null}
from agentx import AgentX
# From the environment (AGENTX_API_KEY, plus AGENTX_API_BASE_URL for self-host)
client = AgentX.from_env()
# Explicit
client = AgentX(
api_key="agtx_local_...",
base_url="http://localhost:4700/api/v1",
)
```
```bash theme={null}
export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
export AGENTX_API_KEY=agtx_local_...
```
## The three modes
| | Default (no auth) | `AGENTX_AUTH=enabled` | + `AGENTX_MULTI_TENANT=true` |
| ----------------- | ------------------------------ | ------------------------------------------------------------- | --------------------------------------------------------------------- |
| For | Local testing, single operator | A self-host **team** sharing one instance | A **multi-tenant** deployment (this is how AgentX's own cloud runs) |
| Dashboard | Auto-connects, no accounts | Email + password session | Email + password session |
| Signup → org | n/a | First user owns the instance; later signups join the same org | **Every signup gets its own organization** + a seeded default project |
| Teammates | n/a | Just sign up | **Invited** (Settings → Team), roles: owner/admin/member |
| LLM provider keys | Env or Settings | Env or Settings, instance-wide | **Per-organization** only - the env is never used for tenant traffic |
| Pricing catalog | Instance-wide | Instance-wide | Global read-only defaults + per-org additions |
| SDK / CI / OTel | `x-api-key` | `x-api-key` (unchanged) | `x-api-key` (unchanged) |
## Multi-user mode: `AGENTX_AUTH=enabled`
For a shared instance, start the engine with `AGENTX_AUTH=enabled`. This switches the dashboard
from the connect screen to real accounts:
* **First boot** shows an owner-setup screen - the first account created becomes the
organization owner and claims the existing projects. Later sign-ups get NO membership on
their own: teammates join by accepting an invitation (Settings > Team), so an exposed port
cannot hand your projects to a stranger who self-registers. Set `AGENTX_OPEN_SIGNUP=true` to
restore auto-join on a closed network.
* **Dashboard requests** ride the session cookie; project keys are handed out through the
session-guarded `/projects` listing after sign-in, never anonymously.
* **SDK / CI / OTel callers** are unchanged - they still authenticate with a project API key
from **Settings → API access**.
## Multi-tenant mode: `AGENTX_MULTI_TENANT=true`
Add `AGENTX_MULTI_TENANT=true` (with auth enabled) for a deployment where strangers sign up -
each signup creates its **own organization** with a seeded default project, and organizations
are hard-isolated: separate projects and data, separate LLM provider keys (Settings → LLM
Providers writes the org's own row; the process env is never used for tenant judge calls), and
separate pricing-catalog additions.
**Teams and invitations** (Settings → **Team**): owners and admins invite teammates by email -
creating an invitation returns a link to send; the teammate signs in (or signs up) with the
invited email and the link adds them to the organization. Invitations are single-use, bound to
the invited email, and expire after 7 days. The owner can't be removed; admins can invite and
remove members.
Set `AGENTX_PUBLIC_URL` so invitation links carry your real domain.
## Email, password reset, and social sign-in
All of these are optional and advertised to the dashboard through `/auth/config`, so the
sign-in screen only shows what the engine actually supports:
* **Email delivery**: configure `AGENTX_RESEND_API_KEY` (Resend) or `AGENTX_SMTP_URL` (any
SMTP server), plus `AGENTX_EMAIL_FROM`. With a transport configured, invitation emails are
sent automatically and the sign-in screen gains a **Forgot password?** link that emails a
reset link (`/reset-password` in the dashboard).
* **Email verification**: add `AGENTX_REQUIRE_EMAIL_VERIFICATION=true` to make new accounts
verify their address before the first sign-in.
* **Social sign-in**: set `AGENTX_GOOGLE_CLIENT_ID`/`AGENTX_GOOGLE_CLIENT_SECRET` and/or
`AGENTX_GITHUB_CLIENT_ID`/`AGENTX_GITHUB_CLIENT_SECRET` to add "Continue with Google/GitHub"
buttons. Register the OAuth callback as
`/api/v1/auth/callback/`.
* **Enterprise SSO (generic OIDC)**: set `AGENTX_OIDC_ISSUER`, `AGENTX_OIDC_CLIENT_ID`, and
`AGENTX_OIDC_CLIENT_SECRET` to add an SSO button (label it with `AGENTX_OIDC_NAME`, e.g.
"Okta"). Works with any IdP that serves OIDC discovery - Okta, Microsoft Entra ID, Auth0,
Google Workspace, Keycloak. Register the callback as
`/api/v1/auth/callback/oidc` - the same shape as the social providers.
(Deployments upgraded from engines older than better-auth 1.7 must update the redirect URI
registered with their IdP: it used to be `/api/v1/auth/oauth2/callback/oidc`.) SAML and SCIM
are not supported; OIDC is the supported enterprise door.
## Operating a multi-tenant deployment
* **Quotas**: `AGENTX_QUOTA_JUDGE_CALLS_PER_DAY` caps judge LLM spend (per organization in
multi-tenant mode) and `AGENTX_QUOTA_TRACES_PER_DAY` caps root-trace ingest per project.
Hitting the trace quota returns a `429` naming the limit; a capped judge call degrades like
a judge outage (the affected result is stored as skipped, with the quota message as its
justification). Both reset at midnight and are unlimited when unset.
* **Admin overview**: set `AGENTX_ADMIN_TOKEN` and call `GET /api/v1/admin/overview` with an
`x-admin-token` header for per-organization members, projects, and 24-hour judge/trace usage.
The endpoint 404s when the token is unset.
* **Organization deletion**: an owner can delete their organization from the API
(`DELETE /api/v1/auth-org/organizations/:orgId` with the org name as confirmation) - every
project and all of its data go with it. User accounts survive; in multi-tenant mode a
returning orphaned account simply gets a fresh organization.
### Putting it together
A complete multi-tenant cloud posture is just environment variables on the same engine binary
or Docker image the single-tenant install uses:
```bash theme={null}
AGENTX_AUTH=enabled
AGENTX_MULTI_TENANT=true
AGENTX_PUBLIC_URL=https://eval.example.com
AGENTX_DB_URL=postgres://... # Postgres recommended for a public deployment
AGENTX_RESEND_API_KEY=re_... # or AGENTX_SMTP_URL
AGENTX_EMAIL_FROM="AgentX "
AGENTX_REQUIRE_EMAIL_VERIFICATION=true
AGENTX_GOOGLE_CLIENT_ID=... # optional social sign-in
AGENTX_GOOGLE_CLIENT_SECRET=...
AGENTX_QUOTA_JUDGE_CALLS_PER_DAY=500
AGENTX_QUOTA_TRACES_PER_DAY=10000
AGENTX_ADMIN_TOKEN=
```
Related engine variables (`AGENTX_PUBLIC_URL`, `AGENTX_TRUSTED_ORIGINS`, the mailer, quota,
and admin variables) are listed in [Configuration](/self-host/configuration).
## Security model
A project API key grants full access to that project's data - treat it like any other secret:
environment variables and CI secret stores, never source control. In default mode the engine
trusts the network boundary ("if you can reach the port, you hold a key you were given"), so
bind it to localhost or a private network, or turn on `AGENTX_AUTH=enabled` before exposing it
more widely.
# Concepts
Source: https://developers.agentx.so/concepts
The core vocabulary - traces, sessions, scorers, signals, datasets, runs, and how they relate
Every capability shares this vocabulary. Each term is defined once, in the table for the area
that owns it - skim the tables, then follow the links for the full guides.
![PLACEHOLDER: entity-relationship style diagram. Left: Agent emits Traces (with child Spans and Tool calls), Traces group into Sessions. Middle: Patterns and Online Evaluators read Traces/Sessions and raise Signals. Right: Datasets (Questions + criteria) drive Evaluation Runs producing Results with ratings. Bottom: Prompts and Tool Schemas registries receive evidence from Signals and Results.]()
## Tracing
| Term | Meaning |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Trace** | One agent interaction: a root record with input, output, latency, model, tokens, and tool calls. See [Tracing](/sdk/tracing). |
| **Span** | One step inside a trace - a graph node, LLM call, tool execution, or retrieval - linked by `spanId`/`parentSpanId` into a tree. The trace dialog's Timeline and Graph views render this tree. See [Span kinds](/trace/span-kinds). |
| **Session** | The conversation a trace belongs to: every trace sharing a `session_id`. Sessions get their own Observe view, turn counts, a conversation-level coherence score, and [session-scoped judging](/monitor/session-evaluation) once the conversation goes idle. See [Sessions](/trace/sessions). |
| **Tool call** | A recorded tool invocation (name, arguments, result, success flag). Feeds tool-quality metrics, the Tool-failure check, and [trajectory matching](#trajectory-match). |
| **Agent** | The named producer of traces. Auto-registered on first ingest; carries a monitoring profile and a healthy rate in the Agents tab. |
## Monitoring
| Term | Meaning |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Scorer** | Anything that can pass judgment on traffic, managed from the one [Scorers tab](/monitor/scorers): patterns, LLM judge scorers, and custom scorers. All opt-in - nothing is judged until a scorer is enabled. |
| **Pattern** | A deterministic scorer: one of the six shipped templates (secrets, PII, prompt-injection echo, profanity, refusal / non-answer, malformed JSON) or a [rule you author](/monitor/patterns) from phrase, regex, and semantic (LLM-judged) conditions. Polarity is `failure` (raises an issue) or `proper` (logs a healthy tally). |
| **Operational outcome** | NOT a scorer: a fact the trace itself recorded - an error, a failed tool call, an empty response. Always classified into the KPI failure metrics, never raises a triage signal, nothing to enable. |
| **Signal** | One deduped finding: repeated matches of the same scorer for the same agent accumulate on one row (occurrence count, last-seen) instead of flooding the list. [Triage](/monitor/review-queue) with statuses, archive or resolve what's done - a re-firing archived or resolved signal reopens automatically. |
| **LLM judge scorer** | ONE judge entity usable everywhere: a rubric (criteria + judge prompt + judge model + tool-context level), an offline profile (dataset-run grading - the scorer's id IS the `scorer_id` runs take), and an optional [online profile](/monitor/online-evaluators) whose single **Score live traffic** switch is the only gate (per trace, or per session once a conversation goes idle). Low live scores raise signals. (Its halves are "evaluation settings" and "online evaluator" in the legacy SDK/API surfaces - same entity, older names.) |
| **Custom scorer** | Your own logic per sampled trace: an **external** HTTP endpoint returning a verdict, or a **code** scorer - a Python or JavaScript `handler()` you write, run in-engine, scoring 0-1. See [Custom scorers](/monitor/custom-evaluators). ("Custom evaluator" in the SDK/API.) |
| **User feedback** | An end user's thumbs up/down forwarded from your app (`client.feedback.report`) - human ground truth attached to the trace: raises a triage signal on a downvote, drives the Downvote rate KPI, and feeds judge calibration. Not a scorer; nothing to configure. See [User feedback](/monitor/user-feedback). |
| **Topic** | Automatic clustering of traffic by subject: the [Topics tab](/monitor/topics) and the Overview topic map, and the traffic side of [dataset coverage](#insights-and-improve). |
| **Outcome** | A real-world result you report back (`client.outcomes.report`) - refund issued, ticket reopened - used to compare judge verdicts against reality. See [Outcomes](/monitor/outcomes). |
| **Judge calibration** | Overview's scoreboard for the judges themselves: how often automated verdicts agreed with recorded reality (reported outcomes, user votes, human re-scores). See [Outcomes](/monitor/outcomes). |
| **Judge tuning** | Rewriting an online judge's grading criteria from its recorded disagreements with reality - validated by exact re-judging (fixes the cases it got wrong, preserves a control set it got right) before a human publishes. |
| **Model comparison** | An Overview card aggregating quality, cost, and latency per model from real traffic - which model is actually earning its bill. See [Model comparison](/monitor/model-comparison). |
## Evaluation
| Term | Meaning |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Dataset** | A versioned, reusable set of test cases plus scoring configuration (criteria, metrics, code scorers, CI settings). Runs snapshot the dataset version they used. See [Build a dataset](/sdk/evaluations/build-dataset); cases can also be [curated from production traces](/evaluation/datasets-from-production). |
| **Question** | One test case: `query`, optional `expectedResults` (the reference answer), optional per-case `judgeGuideline`, optional [expected trajectory](#trajectory-match), optional smoke-test variants (LLM-paraphrased rewordings to catch phrasing brittleness). |
| **Evaluator config** | Legacy name for an [LLM judge scorer](#monitoring)'s rubric + offline profile (the [evaluation settings](/sdk/evaluations/evaluation-settings) record). Not a separate entity since the unification - managed from the Scorers tab, and its id is the scorer's id. |
| **Evaluation run** | One execution of a dataset against an agent: every output, its scores, the locked dataset version, and the subject description. Statuses: `in_progress` → `completed` or `failed`. |
| **Judge rating** | The LLM judge's 0-10 score with a written justification. When a result links its trace, the judge also sees the execution trajectory - tools called, order, failures - not just the final text. |
| **Similarity metrics** | Optional per-dataset scorers against `expectedResults`: vector (cosine) similarity, Jaccard token overlap, BLEU, ROUGE-L. |
| **Code scorer** | A JavaScript function run in-engine against each result - deterministic checks (format, keywords, tool behavior) that need no LLM. See [Code scorers](/evaluation/code-scorers). |
| **Trajectory match** | A deterministic pass/fail comparing the tool calls a linked trace actually made against the case's `expectedTrajectory` (a `tools` list plus a `mode`): `strict` (same calls, same order), `unordered` (same calls, any order), `superset` (all expected present, extras allowed), `subset` (nothing unexpected, missing allowed). |
| **Model portability** | Replay a captured trace's input against alternative models for a quick cost/latency/quality comparison, without touching your agent. See [Model portability](/evaluation/model-portability). |
| **Evaluation subject** | Metadata describing what was evaluated (`kind`, `displayName`, `framework`, optional `version` for [version comparison](/improve/comparing-versions)). |
## CI/CD
| Term | Meaning |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Gate** | The pass/fail decision for a pipeline: rating floor (`fail_under`) and/or no-regression versus a baseline run. Recorded gates land in the dashboard's CI Gates view. See [CI/CD](/sdk/ci-cd). |
| **Pass rate** | Fraction of cases passing every threshold rule (0.0-1.0). |
| **Threshold** | A per-metric rule on the dataset ("judge rating ≥ 7", "cosine ≥ 0.8") counted into the pass rate. |
| **Fail fast** | Finalize the run as failed on the first threshold violation instead of waiting for remaining cases. |
## Insights and Improve
The old Improve tab merged into **Insights**, which holds two views: **Suggestions** and
**Dataset coverage**. The Prompts and Tools & MCPs registries live under the sidebar's
**Manage** section.
| Term | Meaning |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Suggestion** | One queued improvement proposal in **Insights > Suggestions** (the improvement inbox): a background sweep notices fresh failure evidence on a prompt or tool schema, generates the proposal, runs its baseline-vs-candidate validation, and queues it with the measured verdict attached. Humans review, then publish or dismiss. |
| [**Dataset coverage**](/evaluation/dataset-coverage) | **Insights > Dataset coverage**: does your dataset look anything like your traffic? Joins production topics to dataset cases and reports traffic-weighted coverage, topic breadth, and risk-weighted coverage, plus a probe for whether one specific query (or a pasted list) is covered. |
| **Prompt registry** | Versioned prompts your agent pulls at runtime (`client.evaluations.prompts`), with judge-proposed, human-approved rewrites. See [Prompt Management](/improve/prompt-management). |
| **Tool schema registry** | Versioned tool definitions with failure evidence gathered from traces and playground runs, and "Suggest improvement" proposals. See [Tool schemas](/improve/tool-schemas). |
| **Proposal validation** | Before adopting a proposed change, AgentX runs candidate-vs-current against a dataset and shows the measured difference. See [Validating proposals](/improve/validating-proposals). |
# Errors
Source: https://developers.agentx.so/errors
Error response shapes, status codes, Retry-After behavior, and retry guidance
## Response shapes
Almost every error from the self-host engine is a single-field object:
```json theme={null}
{ "error": "Human-readable message" }
```
Two exceptions:
* **Rate limiter** responses carry the shape the dashboard's interceptor reads:
`{ "statusCode": 429, "message": "Too many requests" }`.
* **Trace ingest validation** (`422`) adds the field-level breakdown:
`{ "error": "Invalid trace payload", "details": { "fieldErrors": ..., "formErrors": ... } }`.
The hosted API returns a structured envelope instead
(`{ "status": "error", "statusCode": ..., "message": ..., "errors": [] }`). The Python SDK
normalizes all of these - `AgentXEvaluationsError` and friends carry the message either way,
so application code never needs to branch on the shape.
## Status codes
| Code | When it occurs | Retry? |
| ----- | ------------------------------------------------------------------------------------------------------- | ------------------------ |
| `400` | Missing/invalid field, batch too large, invalid query parameter | No - fix the request |
| `401` | Missing or invalid `x-api-key`; no session in `AGENTX_AUTH=enabled` mode | No - fix credentials |
| `403` | Authenticated but not allowed (e.g. no organization membership) | No |
| `404` | Resource doesn't exist or belongs to another project; admin routes with `AGENTX_ADMIN_TOKEN` unset | No |
| `409` | Run already in a terminal state; dataset's grading config attached to a live scorer; duplicate model id | No - check state first |
| `422` | Trace ingest payload failed schema validation | No - fix the payload |
| `429` | Rate limit, ingest queue full, or daily quota reached (see below) | Depends - see below |
| `500` | Unexpected server error | Yes, with backoff |
| `503` | Trace storage unavailable - the span was **not** stored | Yes, after `Retry-After` |
## The three flavors of 429
1. **Rate limiting** (`{ "statusCode": 429, "message": "Too many requests" }`): per-IP,
per-minute ceilings - 120/min on credential/auth routes, 6000/min on data-plane routes
(tunable via `AGENTX_RATE_LIMIT_CREDENTIAL` / `AGENTX_RATE_LIMIT_DATA_PLANE`;
`AGENTX_RATE_LIMIT=off` disables). Standard `RateLimit` headers (draft-7) are sent. Retry
with backoff.
2. **Ingest queue full** (`{ "error": "Ingest queue is full - retry with backoff (Retry-After: 1s)." }`):
explicit backpressure on `POST /ingest/traces`; nothing was stored. Honor the
`Retry-After` header (1s) and redeliver - span ids keep the redelivery idempotent.
3. **Daily trace quota** (`{ "error": "Daily trace quota reached (...)" }`):
`AGENTX_QUOTA_TRACES_PER_DAY` caps root-trace ingest per project; resets at midnight, so
retrying before then won't help. (The judge-call quota,
`AGENTX_QUOTA_JUDGE_CALLS_PER_DAY`, never returns a 429 - a capped judge call degrades
like a judge outage: the affected result is stored as `skipped` with the quota message in
its `justification`.)
## Common cases
**`401` on every request (self-host)** - the key is wrong or absent. Copy the
`Default project API key: agtx_local_...` line from the engine's startup log; the dashboard
prompts for it on the connect screen, SDK/CI callers set `AGENTX_API_KEY`. The body is always
`{ "error": "Invalid or missing API key" }`. In `AGENTX_AUTH=enabled` mode, dashboard routes
want a signed-in session instead - see [Authentication](/authentication).
**`409` from `POST /runs/:id/results`** - the run is already in a terminal state
(`completed` or `failed`); the body is `{ "error": "Run is already in a terminal state" }`.
Check `GET /runs/:id` before submitting more batches. Note that `POST /runs/:id/finalize`
itself never 409s: it is idempotent on completed runs and returns `status: "failed"` (not an
error) for failed ones.
**`400 "Batch size must not exceed 10"`** - result submission is capped at 10 per batch. The
SDK's `execute()` batches for you; hand-rolled callers should chunk.
**`503` from `POST /ingest/traces`** - the telemetry store is down or the disk is full; the
span was not stored. The response carries `Retry-After: 2`. Redeliver with the same `spanId`.
**`null` ratings instead of errors** - a judge that cannot score (missing judge provider key,
provider outage) does not fail the `/results` call: the result is stored with
`status: "skipped"`, `rating: null`, and the reason in `justification`, and it shows up in
`liveStatistics.skippedCount`. Check `skippedCount` before trusting an average.
## Retrying
* Safe on `500`, `503`, and rate-limit/queue-full `429`s, with exponential backoff (honor
`Retry-After` when present). Quota `429`s reset at midnight - don't spin on them.
* Trace ingest is idempotent when a `spanId` is supplied - replaying the same span stores
nothing new and triggers no duplicate judging. Importers (`agentx-moveworks`,
`agentx-databricks`) rely on exactly this.
* Result submission is idempotent per `idempotencyKey` (the SDK sends one automatically), and
the engine treats a racing duplicate as a duplicate, never an error - so a retried batch
never scores the same case twice. Still, avoid blind transport-level retries of `/results`:
scoring is synchronous and slow, and a retry fired while the first attempt is still scoring
just wastes a request.
# Code Scorers
Source: https://developers.agentx.so/evaluation/code-scorers
Grade eval results with exact JavaScript, alongside the judge and similarity metrics
Evaluate's judge scoring and similarity metrics (vector/Jaccard/BLEU/ROUGE) cover most grading needs, but sometimes what you want to check is exact code, not a model's opinion - did the output actually call this function, is this valid JSON, does the response stay under N words. Datasets and LLM judge scorers (their offline eval config) both support attaching one or more **code scorers**: plain JavaScript functions, run in-process (`node:vm`, sandboxed - no network, no filesystem, a 3-second timeout) against each result's `input`/`output`/`expected` - plus `toolCalls` (the linked trace's recorded tool calls) when the result carries a `trace_id`, so a scorer can assert on tool behavior. They run *after* the judge and similarity metrics, so a scorer also receives `scores` - the result's other verdicts - and can combine them (see below). Live traffic has a counterpart - the [code scorer](/monitor/custom-evaluators) kind on the Scorers page - but the shapes differ: an offline code scorer is a synchronous JavaScript function body like the ones below, while a live code scorer defines an async `handler()` and may also be Python.
From the dashboard: attach them in the dataset editor (opened from the sidebar's Manage → Datasets) or in a judge scorer's **Offline runs** step. Each one gets a name and a JavaScript function body.
From the SDK (dataset builder, so the scorer is versioned with the dataset it guards):
```python theme={null}
dataset = client.evaluations.datasets.builder(
name="Support gate",
code_scorers=[{
"name": "mentions no competitor",
"code": "const bad = /rivalcorp/i.test(output); return { score: bad ? 0 : 1 };",
}],
).add_case(query="...", expected_results="...").publish()
```
```js theme={null}
const wordCount = output.trim().split(/\s+/).length;
if (wordCount <= 60) {
return { score: 1, reasoning: `Concise: ${wordCount} words.` };
}
return { score: 0.5, reasoning: `Wordy: ${wordCount} words (over the 60-word guideline).` };
```
Return a bare number (0-1) or `{ score, reasoning }`. A scorer that throws, times out, or returns something unexpected degrades to `{ score: null, error }` for that one result - it never blocks the judge rating or any other score from being computed.
## Combining judges and metrics into one final score
Because code scorers run after the judges and metrics, each one receives a `scores` object with the result's other verdicts:
```ts theme={null}
scores = {
rating: number | null, // primary judge, 0-10
judges: { [scorerName]: number | null }, // additional judge scorers, 0-10 each
vectorSimilarity: number | null, // cosine, 0-1 (null when not enabled)
jaccardSimilarity: number | null,
bleuScore: number | null,
rougeScore: number | null,
}
```
Direction check before weighting: all four built-in metrics are **higher-is-better** (1 = identical to expected, 0 = nothing in common), and judge ratings are usually higher-is-better too - but a judge whose rubric measures a *bad* quality (a "Toxicity" judge where 10 = worst) must be inverted before it joins the blend (`1 - value / 10`). The dashboard template has a `higherIsBetter` flag per entry for exactly this.
That makes a custom weighted final score a few lines - your own blend of judge verdicts and deterministic metrics:
```js theme={null}
// 50% primary judge, 20% cosine, 30% ROUGE - renormalized over whatever is available.
const WEIGHTS = { rating: 0.5, vectorSimilarity: 0.2, rougeScore: 0.3 };
if (!scores) return { score: null };
let total = 0, used = 0;
if (scores.rating != null) { total += (scores.rating / 10) * WEIGHTS.rating; used += WEIGHTS.rating; }
for (const m of ["vectorSimilarity", "rougeScore"]) {
if (scores[m] != null) { total += scores[m] * WEIGHTS[m]; used += WEIGHTS[m]; }
}
return used > 0 ? { score: total / used } : { score: null };
```
The dashboard's code scorer editor ships this as the **Weighted final score** template (plus **Worst judge wins**, which takes the lowest verdict across the primary and every additional scorer). On a [multi-judge run](/sdk/evaluations/quickstart#multiple-judge-scorers-per-run), `scores.judges` carries each additional scorer's rating by name, so per-judge weights work too. `scores` is `undefined` on surfaces that don't score judges first - always null-check it.
From the SDK, attach scorers when building a grading config:
```python theme={null}
settings = client.evaluations.settings.builder(
name="Strict grading",
acceptance_criteria="...",
code_scorers=[
{
"name": "conciseness",
"enabled": True,
"code": (
"const wordCount = output.trim().split(/\\s+/).length;\n"
"return wordCount <= 60"
" ? { score: 1, reasoning: `Concise: ${wordCount} words.` }"
" : { score: 0.5, reasoning: `Wordy: ${wordCount} words.` };"
),
}
],
).publish()
```
## Reading them back
Code scorers are fully retrievable through the SDK - both the stored configuration and the per-result outputs:
```python theme={null}
# The stored configuration, from either home it can live in:
scorer = client.monitor.judge_scorers.get(scorer_id)
scorer.code_scorers # [{ "name": "Final score", "code": "...", "enabled": True }, ...]
dataset = client.evaluations.datasets.get(dataset_id)
dataset.code_scorers # same shape; None when the dataset has none
# The per-result outputs, one row per enabled scorer:
run = client.evaluations.run(...).execute(my_agent).finalize()
for row in run.results():
row.code_scorer_results # [{ "name", "score", "reasoning"?, "error"? }, ...]
```
`datasets.get()` also round-trips them through `import_dataset()`, so copying a dataset carries its scorers along. Remember which home applies at run time: a run graded by an LLM judge scorer uses **that scorer's** code scorers; only a run with no scorer selected falls back to the dataset's own.
Each result's scorer rows come back via [`run.results()`](/sdk/evaluations/analysis-report#per-result-rows-without-an-analysis) as `codeScorerResults`. The runnable versions are `sample-scripts/selfhost_demo/03_evaluate_with_a_dataset.py` and `sample-scripts/eval_deep_dive/08_weighted_final_score.py`.
[Expected-trajectory matching](/sdk/evaluations/build-dataset#expected-trajectories) reports
through the same per-result scorer rows: a case with `expected_tools` shows a
"Trajectory match (mode)" row next to your own code scorers, pass/fail with the
expected-vs-actual call lists in its reasoning.
# Dataset Coverage
Source: https://developers.agentx.so/evaluation/dataset-coverage
Does your eval dataset look anything like production? Insights joins classified traffic to your cases and shows which high-traffic topics have no test behind them
Your datasets say what you test. Production says what actually happens. **Dataset coverage**
(Governance > Insights > Dataset coverage) is the join: it groups classified production traffic
into topics, measures how deeply your dataset cases cover each topic, and ranks the gaps by
what closing them is worth.
## Prerequisites
* **Topics classification on** (Settings > Monitoring Defaults) - coverage is measured against
the topics Monitor derives from real traffic, so an install with nothing classified shows an
empty state, not zeroes.
* **`OPENAI_API_KEY`** for embeddings. Without one the page still works but runs in a labelled
**Approximate** mode: coverage falls back to counting cases against a per-topic target
instead of measuring depth, and off-map detection is disabled (lexical matching cannot prove
a case matches nothing).
## The three headline numbers
| Card | What it answers |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Traffic-weighted coverage** | Of the classified traffic, how much has a test behind it. Weighted by topic traffic share, so covering a busy topic moves it more than covering a rare one. |
| **Topics at target** | How many topics meet their case target (which scales with traffic and observed risk), with the covered / underrepresented / missing split. |
| **Risk-weighted coverage** | Of the traffic that already fails (issues, negative sentiment), how much is tested. The gap between this and traffic-weighted coverage is the useful finding: "you test what is common, not what is dangerous." |
A fourth card, **Cases off the map**, lists dataset cases matching no topic production asks -
candidates to retire (each opens with its expected results and nearest topic, so retiring is a
read decision, not a guess).
## Coverage is depth, not case count
A topic's coverage is a facility-location value over the topic's real traces: each production
trace is scored against its nearest case, so **near-duplicate cases add nothing** and the
number cannot be inflated by generating copies. A topic whose traces carry no embeddings falls
back to case-count-against-target for that topic alone, and says so ("cases against target -
no depth available").
The "covered" threshold is the same similarity band `addCaseToDataset` uses for dedupe -
covered literally means *the dataset would reject this query as a duplicate*.
## Working the queue
The topics table ranks by **priority** = traffic exposed x (1 - coverage) x risk
amplification - a covered topic is worth nothing to work on however busy, and a risky topic
outranks a healthy one carrying several times its traffic. The **Fix next** chips surface the
top three.
Selecting a topic opens its detail rail:
* the four per-topic numbers (traffic with trace/session counts, cases against target,
coverage with its basis, observed risk with its issue/negative split),
* **Production traces in this topic** - the newest real requests with the classifier's verdict,
so a gap is judged by reading what production actually asked,
* **Suggested action** with two buttons: **Generate N cases from traces** creates candidate
cases from the topic's own traces through the standard curation path (deduped, versioned,
`expectedResults` deliberately left for you to write - a generated case is a scaffold to
review, never a fabricated assertion), and **Write one manually** opens the dataset picker.
## The probe: "is this already tested?"
The probe card answers coverage for one specific query - or a pasted list (a launch spec, a
support macro export) as a pre-launch gate. Verdicts distinguish a **real gap** (production
asks this, nothing tests it) from **untested and unasked** (nothing tests it, but nobody asks -
deliberately neutral, because writing that test is work nobody needs).
## The coverage map
The **Map** tab draws production traffic and your dataset cases in **one picture**: a joint UMAP
projection in **question space** - a trace's input-only embedding against a case's query-only
embedding - so an identical question from both sources lands in the same spot. (The coverage
table's topic matching runs in interaction space, `input+output` vs `query+expected`; that is
the right space for "is this behavior tested" but the wrong one for this picture, since it
separates identical questions whenever the actual answer differs from the expected one.)
Blue dots are classified production traces; orange dots are dataset cases. A translucent
**density cloud** glows around the points in each source's color - clouds emerge only where
points actually cluster, blue and orange blend where dataset cases sit on production traffic,
and a blue region with **no orange tint is the gap**, visible without reading a number.
Reading aids:
* **Source chips** (Both / Production / Datasets) filter the view; the clouds follow.
* The **topic legend** shows `production · dataset` point counts per topic - `34 · 0` is an
untested topic before you have read the chart at all - and hovering a topic spotlights its
points and clouds while dimming everything else.
* **Unmatched cases** (tests no production topic claims) get a dashed chip; they are the same
cases the Off-map tab lists.
* The map inherits the page's date range and dataset filter; cases or traces still warming
show as an "N still indexing" chip rather than silently missing (traces classified before the
question-space column existed backfill in bounded batches as the map is viewed).
Two honest limitations: UMAP coordinates carry no absolute meaning (only relative distance
does, and the layout changes between fetches), and the map requires embeddings on both sides -
unlike the coverage table it has no lexical fallback, because a position IS a similarity claim.
## Scoping and export
The date range (presets from 24 hours to a year, or a custom range; the default is the past
30 days) and the dataset filter scope every number on the page, the map included. **Export**
writes exactly the rows on screen as CSV; the filename is prefixed `provisional-` when the
numbers are still a floor (case embeddings indexing) or `approximate-` when they are lexical.
## API
* `GET /api/v1/insights/coverage?window=30d&datasetId=...` - the full coverage result.
* `GET /api/v1/insights/coverage/map` - the coverage map (same scoping parameters; a separate
route because the joint UMAP fit is heavier than the coverage sweep, so you only pay for it
with the Map tab open).
* `POST /api/v1/insights/topics/curate` with `{ topic, datasetId, window?, limit? }` - the
"Generate N cases from traces" action (returns `{ added, duplicates, considered }`).
* `POST /api/v1/insights/probe` and `/probe/batch` - the probe.
Both GET routes accept either `window` (`24h` / `7d` / `30d`) or an explicit custom range as
`from` and `to` (epoch milliseconds, `to > from`, capped at one year) - the same range override
every monitoring dashboard route takes. `datasetId` and `datasetIds` are interchangeable, each
accepted repeated or comma-separated.
# Datasets from Production
Source: https://developers.agentx.so/evaluation/datasets-from-production
Turn a real trace, session, or signal into a golden dataset case in two clicks
Golden datasets shouldn't only be hand-authored: the best regression tests are the exchanges your agent actually got wrong. AgentX turns any production trace or session into a dataset case, so every flagged failure can become a permanent test - the flywheel that keeps your eval suite honest about what production actually looks like.
## Where to find it
* **Any trace**: open the full trace dialog (Observe → Live Traces → click a trace → **Open trace**, or click through from a signal's **View trace**) → **Add to dataset** in the dialog header.
* **Any session**: open a session's detail view (Observe → Sessions) → **Add to dataset**. The whole conversation becomes one multi-turn case: turn 1 is the main question, later turns become follow-up questions - a real failed conversation becomes a multi-turn regression test the runner already knows how to execute.
* **Any coverage gap**: Insights → **Dataset coverage**'s "Generate N cases from traces" action fills a thin topic through this same path - real traces from that topic, same dedupe, expected results left for a human to write.
## The flow
1. **Preview**: AgentX extracts the user question(s) from what was actually sent - for traces carrying threaded conversation history, each turn's question is that turn's last user message, not the whole history.
2. **Edit**: every question is editable, and each turn shows what the agent actually replied (or how it errored) for context. The expected answer is yours to write - **Suggest** drafts a corrected reference answer with one judge call, as a starting point, never a final answer.
3. **Pick a dataset and add.** The case lands in the dataset's `questions` with a normal [version history](/evaluation/version-history) entry.
## Deduplication
Adding the same failure forty times helps nobody, so every add runs three escalating checks:
1. **Same source** - this exact trace or session was already added to that dataset.
2. **Same question** - normalized exact match against existing cases.
3. **Similar question** - embedding similarity against existing cases (needs `OPENAI_API_KEY`; silently skipped without one). Paraphrases ("What is your policy for refunds?" vs "Hi, what's your refund policy?") are caught; related-but-distinct questions ("How long does a refund take?") are not.
A duplicate comes back as a warning with the existing case quoted - **Add anyway** overrides it deliberately.
## Provenance
Cases born from production carry a `source` field (`traceId`/`sessionId`/`signalId` plus when it was added), so you always know which tests came from reality versus imagination. Provenance survives later dataset edits - the editor reads it off each case and writes it back on save.
## From the API
The same three-step contract is scriptable (same routes on both the dashboard's `/evaluate` mount and the SDK-facing `/custom-agent-evaluations` mount):
```bash theme={null}
# Build a case from a session (or pass {"traceId": ...} for a single trace)
curl -X POST http://localhost:4700/api/v1/evaluate/datasets/case-preview \
-H "x-api-key: $KEY" -H "Content-Type: application/json" \
-d '{"sessionId": "my-session-id"}'
# Draft an expected answer from the real (possibly flawed) exchange
curl -X POST http://localhost:4700/api/v1/evaluate/datasets/suggest-expected \
-H "x-api-key: $KEY" -H "Content-Type: application/json" \
-d '{"query": "Where is my order?", "actualOutput": "Let me check.", "error": null}'
# Append the (edited) case; add "dedupe": false to override a duplicate verdict
curl -X POST http://localhost:4700/api/v1/evaluate/datasets//cases \
-H "x-api-key: $KEY" -H "Content-Type: application/json" \
-d '{"case": {"main_question": {"query": "...", "expectedResults": "..."}, "follow_up_questions": [], "source": {"traceId": "..."}}}'
```
The conversation-level surface these cases come from
What production-born cases are for: measuring candidate fixes
## Synthetic cases from a source document
The other direction - no production traffic yet - is covered by **AI-Assisted Dataset Generation**, the dashed panel on the dataset editor's **Import & generate** tab (under "Generate from source text"): paste the source your agent should be graded against (policy text, API docs, an FAQ) and the engine generates grounded test cases - varied phrasing and difficulty, expected results drawn only from the source, and at least one query the source deliberately does not answer (the correct behavior being to say so). Review the checklist and add the keepers; nothing lands in the dataset unreviewed.
Kept cases are appended to the editor's question list - you still review and save the dataset normally.
**Input format: pasted plain text or Markdown only.** There is no document parser - no PDF, Word, or HTML extraction. For a PDF or Word document, copy the text out and paste it; for a web page, paste the readable text rather than the page source (HTML tags are treated as literal text, which wastes the generator's attention). The API takes the same shape: `POST /evaluate/synthesize-cases` with `{ sourceText, count, guidance?, datasetId? }`, where `sourceText` is the raw text and `datasetId` optionally few-shots the style of an existing dataset's cases.
# Head-to-Head Comparison
Source: https://developers.agentx.so/evaluation/head-to-head
Ask which run answered better, instead of trusting two averages a tenth of a point apart
Two runs of the same dataset score 7.4 and 7.6. Did the change help?
Honestly: you cannot tell. LLM judge scores drift when the rubric wording moves, and they bunch
up in the 7-8 band, so a 0.2 gap between two averages is inside the noise. This is the reason
teams stare at a green delta and still ship a regression.
Head-to-head judging asks a different question, the one a person would actually ask: **for this
question, which of these two answers is better?** A preference between two concrete answers is
far more stable than either answer's absolute score, and it is directly the claim you want to
make when shipping a change.
## Run one
```python SDK theme={null}
comparison = client.evaluations.compare_pairwise(
candidate_run_id, # run A - the new version
baseline_run_id, # run B - what it has to beat
both_orders=True, # judge each pair twice, sides swapped
)
print(comparison.summary.winner) # "a", "b", or "tie"
print(comparison.summary.a_wins, comparison.summary.b_wins, comparison.summary.ties)
print(comparison.summary.flip_rate)
```
```bash REST theme={null}
curl -X POST "$AGENTX_URL/api/v1/evaluate/runs/pairwise" \
-H "x-api-key: $AGENTX_API_KEY" \
-H "content-type: application/json" \
-d '{"runAId": "", "runBId": "", "bothOrders": true}'
```
Both runs must be of the same dataset - a per-question comparison needs the same questions. Run A
is the candidate by convention, so "A wins" reads as "the new version won".
In the dashboard, the same thing lives under **Head to head** in the version comparison dialog,
directly beneath the two averages it exists to second-guess. Nothing judges on open: the panel
shows the last comparison for that pair, and judging again is a click, because every case costs a
judge call.
## Position bias, and why `both_orders` matters
LLM judges favor whichever answer they read first. Left alone, that bias decides comparisons, and
you would never see it - the result looks like a clean verdict.
AgentX designs against it rather than mentioning it:
* The judge is shown "Answer 1" and "Answer 2". It never learns which run is the candidate.
* The presentation order **alternates case by case**, so a biased judge cannot favor one run
across the whole batch.
* `both_orders=True` judges every pair **twice with the sides swapped**. If the winner reverses,
the verdict measured position rather than quality: that pair is recorded as a **tie**, and the
batch reports its **flip rate**.
A high flip rate does not mean the comparison narrowly passed. It means the comparison is
inconclusive, and both the dashboard and `assert_pairwise` say so rather than reporting the win
they would otherwise show.
`both_orders` doubles the judge cost. That is the price of knowing whether the result is real.
## Fail a build on it
```python theme={null}
from agentx.testing import assert_pairwise
def test_new_prompt_is_actually_better():
comparison = client.evaluations.compare_pairwise(
candidate_run_id, baseline_run_id, both_orders=True
)
assert_pairwise(
comparison,
must_win=True, # a tie fails: "no worse" is not the claim being made
max_losses=2, # ...and it must not have broken more than 2 cases winning
max_flip_rate=0.2, # ...on a comparison that is not just position bias
)
```
Each check catches something a single average cannot:
| Check | Catches |
| --------------- | --------------------------------------------------------------------------------- |
| `must_win` | A change that cannot beat what it replaces. A tie is a failure, not a pass. |
| `max_losses` | A change that lifts the average by improving easy cases while breaking hard ones. |
| `max_flip_rate` | A comparison decided by presentation order, which should never read as a pass. |
A comparison run without `both_orders` has no flip rate to check, so `max_flip_rate` is skipped
rather than passing on a fabricated zero.
## What is graded
The head-to-head uses the same material a normal run of that dataset uses: its evaluation
criteria and each case's expected result as the reference answer. This matters more than it
sounds. Without the reference, a confidently vague answer ("Maybe, it depends") can beat a
correct specific one, because the judge has no ground truth to check against.
Override the criteria per comparison when you want to compare on one axis:
```python theme={null}
client.evaluations.compare_pairwise(
candidate_run_id, baseline_run_id,
criteria="Which answer is more concise while still covering the policy?",
)
```
## Reading the result
```python theme={null}
for case in comparison.cases:
print(case.query, case.winner, case.presented_first, case.justification)
```
Each case records which side the judge read first, so any verdict can be audited after the fact.
Cases the two runs do not share - one side produced no answer, or the batch hit the 100-case
per-comparison cap - are listed in `comparison.skipped` with a reason, rather than quietly
dropped in a way that would make a partial comparison look like a sweep.
Past comparisons are stored and included in [backups](/self-host/backup):
```python theme={null}
client.evaluations.list_pairwise(run_a_id=candidate_run_id)
client.evaluations.get_pairwise(batch_id)
```
## When to use which
"Is this good enough to ship?" Use a rating floor - [CI gates](/sdk/ci-cd) and
`min_rating`. A threshold needs a number.
"Is this better than what we have?" Use pairwise. A comparison needs a preference, and
preferences survive judge drift that absolute numbers do not.
Most teams want both: a floor that stops bad releases, and a head-to-head that proves the change
was worth shipping.
# Model Portability
Source: https://developers.agentx.so/evaluation/model-portability
Replay a captured trace against alternative models for a cost/latency/quality estimate
For any captured trace, self-host can estimate how a different model would have handled the same input - cost, latency, and a quality rating, side by side with what your agent actually returned. This is an **input-only replay**, not a full re-run of your agent: self-host doesn't own your agent's code, so it has no guaranteed system prompt, tools, or conversation history to work with - only whatever the trace itself captured.
## What gets replayed
What actually gets replayed depends on how you instrumented tracing. Checked against the real SDK tracer:
* The raw Anthropic client patch (`agentx.integrations.anthropic`) captures the full `messages` array verbatim - multi-turn history is included. (It doesn't currently capture Anthropic's separate `system` kwarg, so the system prompt specifically can be missing even when history isn't.)
* The manual API (`client.tracer.trace(name, input=..., metadata=...)`) is completely free-form - pass `input` as a proper `[{"role": "user", ...}, ...]` message list, or `metadata={"systemPrompt": "..."}`, and self-host uses exactly that.
* The higher-level framework integrations (OpenAI Agents SDK, LangChain) flatten down to plain text by default - a trace from one of these is closer to single-turn-only unless you also pass richer `metadata` yourself.
Self-host makes a best-effort attempt to reconstruct a real conversation from whatever shape your trace actually has (recognizable message array → use it; plain text → single user turn; a `systemPrompt`/`system`/`instructions` key in `metadata` → prepended as the system prompt) - the same defensive, try-every-known-shape approach the OTel ingestion path already uses. **Tool-calling is never reproduced**, even if your trace's metadata includes tool definitions - replaying those would mean translating an arbitrary captured schema into each candidate provider's own tool format, real separate work this doesn't attempt. If your trace's metadata has tool definitions, they're shown to you in the dialog for transparency, just not sent to candidate models.
## Running a comparison
The comparison runs from the SDK or the REST API against any trace id (the dashboard trigger was retired, but the engine surface is unchanged):
```python theme={null}
result = client.monitor.run_model_portability(trace_id, ["gpt-4o-mini", "claude-haiku-4-5"])
print(result["reconstructed"]) # exactly what was sent: system prompt if found, each turn,
# and a note if it fell back to plain text
for r in result["results"]:
print(r["model"], r["rating"], r["estimatedCostUSD"], r["latencyMs"])
```
Every model, including the original captured output, is judged with the same no-ground-truth rubric ("rate this response on its own merits, 0-10") so ratings are directly comparable. Cost is estimated from real measured token usage against pricing you control - see below. A candidate needs its provider's key configured on the engine; a model that fails (missing key, rate limit) never blanks out the rest of the comparison - its row comes back with a null output and the reason (including the exact key to set) in its `error` field. Nothing about the comparison itself is persisted - it's computed fresh each time you run it.
## The pricing catalog
The candidate model list and \$/M-token prices are dashboard-managed, not hardcoded - click **Manage models** (next to the judge-model picker in Settings and in the judge scorer editor) to add, edit, or remove entries. A small curated set is seeded the first time the engine starts, so there's something to compare against out of the box; edit or delete those freely after that, or add your own - seeded entries you remove stay removed, except three catalog staples (`gpt-4o-mini`, `gemini-2.5-pro`, `gemini-2.5-flash`) that are re-added on boot whenever missing. Prices are approximate and point-in-time by nature (no live pricing API exists or should exist here), so keep them updated against each provider's current pricing page if you're relying on the cost estimate for a real decision.
### Cache-aware cost
This same pricing catalog also drives the estimated-cost figures on traces themselves, not just the portability comparison. When the SDK's underlying provider call reports prompt-caching token counts - Anthropic's `cache_creation_input_tokens`/`cache_read_input_tokens`, OpenAI's `prompt_tokens_details.cached_tokens`, Gemini's `cached_content_token_count` - those are captured as a subset of the trace's input tokens and priced separately from a regular input token if you've set optional **$/M cache-read tokens** / **$/M cache-write tokens** rates on that model in **Manage models**. Leave either blank and it falls back to the model's regular input rate - cache-aware pricing is opt-in per model, and a model with no cache rates configured prices exactly as it always has.
# Playground
Source: https://developers.agentx.so/evaluation/playground
Test prompts, models, and tools against real dataset cases interactively
![PLACEHOLDER: entity-relationship style diagram. Left: Agent emits Traces (with child Spans and Tool calls), Traces group into Sessions. Middle: Patterns and Online Evaluators read Traces/Sessions and raise Signals. Right: Datasets (Questions + criteria) drive Evaluation Runs producing Results with ratings. Bottom: Prompts and Tool Schemas registries receive evidence from Signals and Results.]()
Before committing to a full dataset run, test a prompt against real models and real test cases interactively - modeled on Braintrust's and OpenAI's own playgrounds. Governance → **Playground**, its own top-level tab:
* Edit a multi-turn prompt - system message, optional few-shot user/assistant example turns - freely, or load a saved prompt's current version from the [Prompt Registry](/improve/prompt-management) as a starting point.
* Select one or more OpenAI, Anthropic, and/or Gemini models, from the same dashboard-managed model catalog [Model portability](/evaluation/model-portability) uses.
* Pick an existing dataset and check which of its questions to include.
* **Run** (the button reads "Run N cells" once models and cases are picked) fires the whole (question × model) grid for real - each cell independently shows the actual output, latency, tokens, and estimated cost, plus a judge rating whenever that question has an expected answer, and each of the dataset's enabled [code scorers'](/evaluation/code-scorers) results alongside it. At most 4 cells run at once (a fixed concurrency cap, so a large grid doesn't trip your provider's rate limits) - the rest queue and fill in as earlier ones finish.
Playground is a scratchpad with a memory: nothing here creates a real eval run or saves a
prompt version on its own, but past runs are kept in **History** (the breadcrumb's Edit |
History toggle, with a Runs | Simulations switch inside) so a refresh doesn't lose your grid,
and past simulations reopen with their full transcript. Copy a result you like into a saved
Prompt or a real dataset run once you're happy with it.
## Scorers: one panel, two roles
The sidebar's single **Scorers** panel covers both grading and monitoring - every
[LLM judge scorer](/monitor/online-evaluators) and custom pattern is one row:
* Exactly one judge carries the **star**: it *grades the run* - reference-based against the
dataset case's expected results, supplying the score, similarity metrics, and repetitions
(the workspace's default scorer is preselected, provisioned automatically on first use).
* **Checked** rows additionally *dry-run* the way live monitoring would: each cell gets a badge
per check - a pattern's reads "Matched" / "No match", a judge's shows its reference-free
rating. Nothing here saves signals or spends beyond the checks you tick. Starring a judge
removes it from the check set - the grader never runs twice.
* Row hover **Edit** and **New judge scorer** open the unified scorer editor without leaving the
Playground; a freshly created judge becomes the grader.
A single query (no dataset case) has no expected answer, so the grader is skipped and checked
judges still score reference-free.
## Tools
The Playground tests prompt and tools together. Add a tool ad hoc, or pull one straight from the [Tools & MCPs registry](/improve/tool-schemas) with the **From Tools & MCPs** picker - name, description, and parameter schema come in as the registry's current published version.
The endpoint URL is **optional**:
* **Blank (simulated)**: the tool is still sent to the model, and its calls return a canned `{simulated: true}` result - so whether the prompt/model *choose* the tool and *form valid arguments* is fully testable with zero infrastructure. The cell's tool-call trace shows the arguments and marks the result simulated, so nothing pretends to be a real lookup.
* **Set**: each call POSTs `{tool, arguments}` to your real service and feeds the actual response back to the model - the full loop, including how the prompt handles real tool outputs.
## Beyond single answers
The grid tests single-shot answers. [Simulate conversation](/evaluation/simulate-conversation)
tests multi-turn behavior with a persona-driven simulated user, and
[Model portability](/evaluation/model-portability) replays any captured trace against
alternative models for a cost/latency/quality comparison.
# RAG Evaluation
Source: https://developers.agentx.so/evaluation/rag
The five RAG metrics, online and offline, with context captured automatically from your traces
RAG failures split into two kinds - the retriever fetched the wrong chunks, or the generator
ignored the right ones - and a single end-to-end score can't tell you which. AgentX ships the
standard component-split metrics as seeded judge-scorer templates (New scorer → Template scorer on the Scorers page), usable
**online** (live traffic) and **offline** (dataset runs) with the same configs.
## The metric pack
| Metric | Judges | A low score means |
| ----------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------- |
| **RAG: Context Relevancy** | Are the chunks on-topic and sufficient for the query? (ignores the response) | Fix chunking / embeddings / top-K |
| **RAG: Contextual Precision** | Are the relevant chunks ranked above the irrelevant ones? | Fix the reranker |
| **RAG: Contextual Recall** | Does the context cover what the *expected answer* needs? (offline - needs a reference) | Retrieval misses source material |
| **RAG: Faithfulness** | Is every claim in the response grounded in the chunks? | The generator hallucinates past its context |
| **RAG: Answer Relevancy** | Does the response actually address the query? | Fix the prompt |
**Add** one from the template picker and it becomes an ordinary judge scorer of your own: tune
its criteria, enable its live scoring, or pass it as `scorer_id` on a run. Deleted ones stay
deleted; a metric added in a release is backfilled once to existing projects.
## Where the context comes from
`{context}` in these judge prompts resolves through the same precedence everywhere:
1. **Explicit** - `metadata.retrievalContext` on a trace, or `retrieval_context` returned by
your agent function in a run (string or list of chunk strings).
2. **Recorded retrievals** - the trace's own retrieval spans: `tracer.trace_retrieval(...)`,
LangChain/LlamaIndex retriever callbacks, OTel retrieval spans. **No caller changes needed**:
if your integration already records retrievals, the judges see the chunks. Custom span names
are fine (`trace_retrieval("kb_search", ...)`) - retrieval spans carry an explicit
[span kind](/trace/span-kinds), so they're recognized regardless of what you name them.
3. **Pinned** (offline only) - the dataset case's static `retrievalContext`.
## Online: score live RAG traffic
Add the metric as your own scorer (Scorers → **New scorer** → **Template scorer** → **Add**),
then enable its live scoring - in the editor's **Live scoring** step, or from the SDK - and
every sampled trace is judged with whatever it actually retrieved:
```python theme={null}
client.monitor.judge_scorers.update(
faithfulness_scorer_id, # your clone of RAG: Faithfulness
online={"enabled": True, "sampleRate": 0.2, "alertThreshold": 5},
)
```
A response claiming a 90-day refund while the retrieved policy chunk says 30 days scores 0,
with the contradiction named in the justification - and raises a signal you can send straight
to a [regression dataset](/evaluation/datasets-from-production).
## Offline: dataset runs with dynamic context
A real RAG agent retrieves per query, so return what it retrieved alongside the answer - the
faithfulness judge grades against *that run's* chunks, not a stale pin:
```python theme={null}
def rag_agent(case):
# Tracing the case links each result to its trace (the View Trace button in the run
# detail table, plus trajectory-aware judging); monitor=False keeps eval traffic out
# of online checks so nothing gets double-judged.
with client.tracer.trace("rag-agent", input={"query": case.query}, sync=True, monitor=False) as span:
with client.tracer.trace_retrieval("kb_search", query=case.query) as r:
chunks = retriever.search(case.query)
r.output = [c.text for c in chunks]
answer = generate(case.query, chunks)
span.output = answer
return {
"output": answer,
"retrieval_context": [c.text for c in chunks],
"trace_id": span.trace_id,
}
client.evaluations.run(
dataset_id=dataset.id,
scorer_id=faithfulness_scorer_id,
).execute(rag_agent).finalize()
```
The explicit `retrieval_context` return is optional here - with the retrieval recorded on the
linked trace, the engine pulls the chunks from its retrieval spans automatically; returning it
just takes precedence. **Contextual Recall** additionally needs `expected_results` on the case
(it attributes the reference answer's claims to the context).
## Deterministic: expected context match (no judge)
The five metrics above are LLM judges - each score is a model call. For retriever *regression*
checks, a case can instead pin the chunks a correct retriever should fetch, and the engine
compares them against what was actually retrieved with **token-level Jaccard similarity** -
deterministic, free, and safe to run on every retriever, chunking, or embedding change:
```python theme={null}
dataset = (
client.evaluations.datasets.builder(name="Retriever regression")
.add_case(
query="What is the refund window?",
expected_retrieval_context=["Refunds are available within 30 days of delivery..."],
)
.publish()
)
```
The same field is editable in the dashboard: open a dataset case and fill **Expected
retrieval context** (blank line between chunks), next to Expected tool calls. Full parameter
reference: [Build a dataset](/sdk/evaluations/build-dataset#expected-retrieval-context).
Each result gets a **Context match (jaccard)** scorer row (0-1) next to any judge scores,
computed from the same actual context the judges see (the result's `retrieval_context`, else
the linked trace's retrieval spans). Jaccard measures token overlap, not meaning - use it to
catch "the retriever stopped returning the right chunk", and the judge metrics for whether
the chunks are *semantically* right.
## The loop
Capture retrievals → RAG pack scores live traffic → low scores raise signals → failing traces
become dataset cases → offline runs guard the fix → [CI gates](/sdk/ci-cd) keep it from
regressing.
## Runnable examples
Four end-to-end scripts in
[`sample-scripts/sdk_rag_samples`](https://github.com/AgentX-ai/AgentX-Sample-Scripts/tree/main/sdk_rag_samples):
online Faithfulness with automatic retrieval-span context, online Context Relevancy catching a
broken retriever (and raising a high-severity signal), offline Faithfulness with dynamic
per-case context, and the deterministic Jaccard retriever regression check.
# Simulate Conversation
Source: https://developers.agentx.so/evaluation/simulate-conversation
A persona-driven simulated user converses with your prompt, model, and tools - judged, recorded, and convertible into regression tests
The [Playground](/evaluation/playground) grid tests single-shot answers; **Simulate conversation** (below its Run button) tests multi-turn behavior. A simulated user - a persona and a goal you write, played by its own model - converses with your current prompt, model, and tools for up to a turn cap (default 5, engine maximum 10):
* The simulated user opens the conversation, reacts in character to each agent reply, and ends the conversation itself: `GOAL ACHIEVED` when the agent genuinely got them there, `USER GAVE UP` when the persona would realistically walk away, or `TURN LIMIT REACHED`.
* The agent side of each turn is exactly a Playground run - same model settings, same tools, including simulated tool results for schema-only tools.
* With a judge scorer starred in the Playground's Scorers panel, the finished transcript is scored 0-10 against its criteria (reference-free - a simulated conversation has no expected answer); the result names the judge that scored it.
* Turns stream into the transcript in real time as each exchange completes, and every finished simulation is kept in the Playground's **History** (Runs | Simulations) for later review or re-running.
The same simulation runs from the SDK - useful for scripted persona sweeps (each call blocks
for the whole conversation, one LLM call per turn plus the closing judgment):
```python theme={null}
result = client.evaluations.simulate_conversation(
model="gpt-4o-mini",
system_prompt=SYSTEM_PROMPT,
persona="An impatient customer whose order arrived three weeks late...",
goal="Get written confirmation of the exact refund amount.",
max_turns=5,
tools=[LOOKUP_TOOL], # optional; simulated results unless an endpoint is set
agent_name="sim-support-agent",
)
print(result["outcome"], result["sessionId"]) # "goal_achieved" / "gave_up" / "max_turns" / "error"
```
(The dialog renders those wire values as GOAL ACHIEVED, USER GAVE UP, and TURN LIMIT REACHED.)
**Recorded as a real session.** Unless you untick "Record as a session", every turn is written through the normal ingest path under one `sim-` session with `metadata.simulated: true` - each turn a real span tree (the model call and any tool executions as timed child spans, so the trace dialog's Execution Timeline works). That means the simulation shows up in Observe → Sessions like any conversation: the built-in [Session Baseline Judge](/monitor/online-evaluators#per-trace-vs-per-session-scope) (once you've enabled it) judges it after it goes idle, session-scoped evaluators score it, and **Add to dataset** works from the session view - so a bad simulated conversation converts directly into regression test cases.
# Version History
Source: https://developers.agentx.so/evaluation/version-history
Every dataset and evaluator edit, saved as a browsable version
Every edit to a dataset or LLM judge scorer is saved as a new version, in a log of its own per entity - a dataset version tracks name, description, questions, and status; a judge scorer version tracks its rubric and offline config (the scorer's version panel lives in the editor's header). Each version is summarized by what changed (`"Updated questions"`, `"Updated acceptance criteria, judge model"`), shown newest-first with a `v{N}` badge, browsable from a **Version history** panel. On any older version, click **Apply this version** to pull its fields back into the form (you still have to hit Save to actually restore it - this doesn't overwrite anything on its own), or **Delete** to remove that version from the log; the current version offers neither. The very first save is always recorded too (`"Created"`), so a freshly-made dataset never shows an empty history. A no-op save (open the editor, change nothing, hit Save) doesn't create a new version - only genuine field changes do.
# Auto-improve
Source: https://developers.agentx.so/improve/auto-improve
Close the loop from confirmed production failures to code fixes - Confirm verdicts accumulate into an improvement group, one report clusters them into issues, and the auto-improve skill applies the fixes to your agent's source
Online evaluation exists to catch your agent failing in production. **Auto-improve** is what
happens after the catch: the failures a human *confirmed* are spent - as one body of evidence -
on an improvement report, and the report is applied to your agent's actual source code by a
coding agent. The loop runs entirely on **online evidence**: live-traffic verdicts a person
vouched for, never offline dataset runs.
## 1 · Confirm failures (Review)
In **Governance → Review → Review signals**, passing a **Confirm** verdict on a signal - a low
judge score or a matched failure pattern - automatically lands the confirmed occurrence in the
**pending batch** ("Confirmed failures"). Confirm *is* the accumulation gesture; there is no
second button, and declining is choosing **Ignore**. The evidence (input, output,
judge rationale, score) is snapshotted at confirm time, so later trace pruning cannot hollow it
out. Re-confirming the same verdict after a reopen never double-counts.
What makes this evidence unusual: every item was flagged by a machine **and** confirmed by a
person. It is the high-precision slice of production traffic - the opposite of auto-selected
"worst examples".
## 2 · Generate a report (Insights → Auto-improve)
**Governance → Insights → Auto-improve** shows the pending batch and its count. **Generate
improvement report** runs one LLM pass (explicit and billed - never implicit) that clusters the
confirmed failures into issues, each with a title, description, recommendation, and its
evidence inline. Generating **seals the batch**: the group becomes that report's permanent
source-case set (renamed with its seal time) and the pending accumulator is cleared - the next
Confirm starts a fresh batch, and the next generate produces a new report from a new group. One
report per group, one group per report. The report is stored under its own id, shown with a
copy button.
From the SDK:
```python theme={null}
groups = client.monitor.improvement_groups.list()
report = client.monitor.improvement_groups.generate_report(groups[0]["_id"])
print(report["_id"], len(report["issues"]))
```
## 3 · Apply the fixes (the auto-improve skill)
The report id is the hand-off to the [AgentX Agent Skill
plugin](https://github.com/AgentX-ai/AgentX-Eval-Skill). In your agent's repo, tell your coding
agent:
```
Use the agentx auto-improve skill on report .
```
The skill fetches the report from the engine (`GET
/agent-monitoring/improvement-reports/:id`), then **triages** each recommendation against your
real source rather than applying it literally - the recommendations were written by a judge
that never saw the code, so they are hypotheses, not instructions. Each issue gets a verdict
(apply / already handled / reject with a reason), and the surviving fixes are made in place.
## Verifying the fix
The same scorers keep watching production. A fix that holds stops re-raising its signals; a
signal you resolved as **Fixed** that fires again reopens as a regression - the engine telling
you the fix did not hold. For a pre-deploy check, curate the confirmed failures into a dataset
and run an offline evaluation against it.
Auto-improve is self-host only, and the report generator uses the engine's default judge
model (override with `generate_report(group_id, model="...")` or the route's `model` field).
# Claude Code
Source: https://developers.agentx.so/improve/claude-code
The AgentX plugin (/instrument, /run-eval, /eval-fix, /auto-improve) and the prompt-improvement skill - drive the whole loop from Claude Code, no engine-side LLM key needed
Two ways to drive AgentX from Claude Code: the **AgentX plugin** (instrument → evaluate → fix
as slash commands, plus [/auto-improve](/improve/auto-improve) for applying confirmed
production failures), and the **improve-prompt skill** (the [Prompt
Management](/improve/prompt-management) loop with Claude's own reasoning standing in for the
server-side judge call).
## The AgentX plugin
One plugin, one loop: get an agent's real runs into AgentX, score them, then turn what they
measure into a code fix. Install it from the plugin marketplace:
```bash theme={null}
claude plugin marketplace add AgentX-ai/AgentX-Eval-Skill
claude plugin install agentx@agentx
```
| Command | What it does |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/instrument` | Sets a Python agent up on AgentX end to end: reads the project API key off the engine, writes `.env.agentx`, installs `agentx-python`, adds one span where the run begins plus framework auto-instrumentation - then proves it by running the agent and reading its own traces back. |
| `/run-eval` | Evaluates that agent against a dataset it can also create (templates, a CSV/file of Q\&A, or cases curated from the agent's own live traces): writes a committed eval harness that calls the agent per case with a linked trace, executes the run, and hands back the score, the browser report, and the `/eval-fix` command. |
| `/eval-fix ` | Triages the evaluation's judge findings against your real source code (never applying code-blind recommendations literally), applies what survives, and re-runs against the same dataset so the before/after comparison means something. |
The seam between the commands is the point: an evaluation result carrying a `traceId` is judged
against the agent's **real execution path**, where one without it is judged on answer text
alone. All three commands talk to a local self-host engine (normally `http://localhost:4700`) -
see [Installation](/self-host/installation) to get one running.
## The improve-prompt skill
[Prompt Management](/improve/prompt-management)'s **Suggest improvement** needs a provider key
(`OPENAI_API_KEY`/`ANTHROPIC_API_KEY`/`GEMINI_API_KEY`) set on the engine, since it makes a
real judge call server-side. If you don't have one configured - or you're already working in
Claude Code anyway - there's a second way in, the same idea as [Langfuse's Claude-skill
prompt-improvement workflow](https://langfuse.com/blog/2026-02-16-prompt-improvement-claude-skills):
Claude's own reasoning stands in for the judge call, so no engine-side LLM key is needed at all.
### Setup
Copy the skill from
[`AgentX-trace-eval/skills/improve-prompt`](https://github.com/AgentX-ai/AgentX-trace-eval/tree/main/skills/improve-prompt)
into `.claude/skills/improve-prompt/` in whatever project you run Claude Code from (your own
agent's repo works fine - the skill only talks to your self-host engine over HTTP). Then, with
your engine running:
```
"Improve the support-agent-system-prompt prompt"
```
### What it does
Claude reads the prompt's real worst-rated eval results straight from your engine
(`GET /api/v1/evaluate/prompts/:id/examples` - the same evidence **Suggest improvement** uses,
just without a judge call in front of it), drafts a full rewrite and explains what changed and
why, and shows you both versions side by side. It never publishes on its own - only once you
say something like "publish it" does it call the same
`POST /api/v1/evaluate/prompts/:id/versions` the dashboard button uses.
It finds your engine automatically: the API key comes straight from `~/.agentx/config.json`
(written the first time you start the engine), and the base URL defaults to
`AGENTX_API_BASE_URL` - the same variable the SDK itself reads - falling back to
`http://localhost:4700/api/v1`.
# Comparing Versions
Source: https://developers.agentx.so/improve/comparing-versions
Run your agent twice, tag each run, let AgentX tell you which version won
AgentX's hosted platform has an "autotune" workflow: propose an instruction change, run it on a candidate branch of the agent's config, compare against the current version, merge if it wins. Self-host can't do the branch/merge/apply part - there's no agent config here for AgentX to own or edit, your agent's code lives entirely outside AgentX. What *is* portable is the comparison: run your agent twice against the same dataset via the SDK - once as-is, once with a change you made yourself - tag each run with a version label, and let self-host tell you which one scored higher.
```python theme={null}
client.evaluations.run(
dataset_id="...",
subject={"kind": "custom_agent", "metadata": {"version": "v2-shorter-prompt"}},
).execute(my_agent_fn)
```
No SDK changes needed - `version` just needs to be a key inside `metadata`.
## Reading the verdict
From the dashboard: the sidebar's **Manage → Datasets** → a dataset's row menu → **Compare versions**. It shows every version that's been run against that dataset (run count, rated count, average rating) and a headline verdict comparing the two most recent versions, the same `candidateAvg >= baselineAvg` check autotune's own validation step uses. Nothing gets merged or shipped automatically - you read the verdict and go edit your own code.
The dialog also includes a **Case by case** drill-down: each version's latest run diffed per question, with rating deltas, regressions highlighted, and both outputs (plus the judge's justification) one click away - the averages say whether a version regressed, this says where.
# MCPs
Source: https://developers.agentx.so/improve/mcp
Register tools from remote MCP servers - listed, reviewed, improved like any tool, and executed over the real protocol in Playground runs
## Registering from a remote MCP server
[Tools & MCPs](/improve/tool-schemas)' **Register tool** dialog has a **Remote MCP** source: enter the MCP name and server URL (plus optional key-value pairs, sent as HTTP headers when connecting - e.g. `Authorization: Bearer ...`), hit **Load MCP server**, and the engine connects (Streamable HTTP, with legacy SSE fallback) and lists the server's tools. Review the list - select, rename, edit descriptions and parameter schemas - then register the keepers, individually or all at once. Each becomes an ordinary tool-schema row whose definition embeds an `mcp` provenance block, and the registry keeps MCP tools recognizable from it: rows are labeled **`mcp name > function name`** (e.g. `paypal > create_invoice`), and the label survives judge-proposed rewrites of the definition. Everything else about the registry applies unchanged: the engine never executes MCP tools in production, it observes your agent's traced calls and improves the definitions. Listing is deliberately forgiving about schema quality - real-world servers ship parameter patterns that strict JSON-schema validators reject, and the engine lists them anyway and lets you clean them up in review.
## Playground execution
Adding an MCP-registered tool via the Playground's "From Tools & MCPs" picker prefills its server URL as the endpoint, and Playground/simulation runs execute the tool's calls over the real MCP protocol (registry browsing stays execution-free - only runs you start execute anything). Works for open and header-free servers out of the box; OAuth-protected servers get a **Connect** button that runs the provider's sign-in popup once and reuses the authorized session for subsequent calls (sliding expiry). Replacing the prefilled URL with your own endpoint switches that row back to the plain POST `{tool, arguments}` contract.
## OAuth-protected servers
Servers that follow the MCP OAuth 2.1 spec (PayPal, Atlassian, ...) work without any manual key setup: when the server answers the first connection with an authorization challenge, the dashboard opens the provider's consent page in a popup. Sign in and approve access there - the popup closes itself and the tool list loads automatically. The engine handles discovery, dynamic client registration, and the PKCE code exchange behind the scenes, holding the resulting tokens in memory only, on a sliding one-hour session that stays alive while it's actually being used (so a Playground run keeps working) and evaporates an hour after the last call; nothing OAuth-related is ever persisted with the registered tools. If your dashboard is served from a domain the engine can't infer, set `AGENTX_PUBLIC_URL` so the OAuth callback (`/api/v1/mcp-oauth/callback`) resolves to a URL the provider can reach.
# Prompt Management
Source: https://developers.agentx.so/improve/prompt-management
A version-scoped prompt registry with evidence-fed improvement proposals
[Comparing versions](/improve/comparing-versions) tells you which one won; Prompt Management (a prompt registry, reached from the sidebar's **Manage → Prompts**) is how self-host closes the rest of the loop without ever touching your agent's code. LangSmith's Prompt Hub and Langfuse's own Prompt Management solve the same "we don't own your agent" problem the same way: become the prompt's source of truth, so your agent pulls a version at runtime, and treat "improvement" as propose → you approve → publish a new version - never a direct edit to your deployed code. Self-host does the same thing:
## Pull prompts at runtime
```python theme={null}
prompt = client.evaluations.prompts.get("support-agent-system-prompt")
prompt = client.evaluations.prompts.get(prompt.id) # by id instead of name works too
# use prompt.text as your own agent's system prompt, however you call your LLM
client.evaluations.run(
dataset_id="...",
subject={"kind": "custom_agent", "metadata": {
"promptName": prompt.name,
"version": f"{prompt.name}@v{prompt.version}",
}},
).execute(my_agent_fn)
```
Create a prompt from the SDK (`client.evaluations.prompts.create(name, text)`) or the dashboard.
## Suggest improvement
From the dashboard: **Manage → Prompts** (self-host; hosted: **Improve → Prompt Management**) → a prompt's row menu → **Suggest
improvement**. It gathers real evidence from two places at once - deliberate Evaluate runs tagged with the prompt's name (as above), and worst-scoring [Online evaluator](/monitor/online-evaluators) ratings from real production traffic, since a live trace can carry the exact same `metadata.promptName` tag a run can:
```python theme={null}
with client.tracer.trace("support-agent", metadata={"promptName": prompt.name}) as span:
... # your agent's own call, using prompt.text as its system prompt
```
### The evidence panel
Both sources are merged, worst-rated first, and load as soon as the dialog opens: this evidence already exists (a plain data read, no judge call), so you don't have to generate a rewrite just to see what's going wrong. Evaluate-run evidence defaults to the *current published version only*, a v3 prompt's rewrite shouldn't get polluted by v1 complaints v2 or v3 may have already fixed, automatically widening to every version if there aren't at least a few examples on the latest one yet. The evidence panel also runs a second, informational judge pass grouping the same examples into a handful of named recurring **failure themes** (e.g. "Curt or unempathetic tone"), so you can see the shape of the problem before reading every individual justification.
### Generate, review, publish
Each example has a checkbox, filterable by source (production monitoring vs. eval dataset runs); only the checked ones feed **Generate suggestion**, so a rewrite can be scoped to a specific failure mode instead of always using everything gathered. The result includes the rewritten prompt, a short overall summary, and an itemized **what changed** list (each entry tagged added, tightened, or removed), plus the judge model actually used. The current and proposed prompts are shown as an editable, line-by-line diff (unified or side-by-side): you can edit the proposed (green) lines directly in the diff before publishing, not just accept it verbatim. Nothing is saved until you click **Publish as new version**. Your agent's next `client.evaluations.prompts.get(name)` call picks up the new version immediately. Tagging your run's `metadata.version` as `@v` (shown above) means the [**Compare versions**](/improve/comparing-versions) dialog already tells you whether the published rewrite actually scored better, no separate comparison view needed.
Before publishing a proposal, [validate it](/improve/validating-proposals): the candidate and the current version run against a golden dataset's cases under identical conditions, and the verdict ships with the published version.
### The same loop from the SDK
Everything above is scriptable - gather the evidence, generate the proposal, and (once a human
has signed off however your process does that) publish:
```python theme={null}
evidence = client.evaluations.prompts.examples(prompt.id) # merged, worst-rated first
proposal = client.evaluations.prompts.propose(prompt.id) # one judge call
print(proposal["revisedText"], proposal["reasoning"])
client.evaluations.prompts.publish_version(
prompt.id,
text=proposal["revisedText"],
reasoning=proposal["reasoning"],
based_on_version=prompt.version,
)
```
The dashboard flow stays the human-review path (editable diff, scoped evidence checkboxes);
the SDK path exists for automation that carries its own approval step - a nightly job opening
a PR with the proposed rewrite, for example.
# Tools
Source: https://developers.agentx.so/improve/tool-schemas
A version-scoped registry for tool definitions, improved from real failures
The prompt isn't the only text you feed an LLM that quietly decides your agent's quality - tool definitions (the name, description, and JSON parameter schema you pass as `tools=[...]`) are the other half, and a vague parameter description causes malformed tool calls the same way a vague prompt causes bad answers. **Tools & MCPs** in the sidebar's Manage section gives them the same registry treatment as prompts: version-scoped, edited in a schema-aware JSON editor, with full version history.
**Suggest improvement** works the same way as a prompt's, fed by tool-specific evidence: recorded tool calls that failed (`success: false`, the same field [`tracer.trace_tool_call`](https://github.com/AgentX-ai/AgentX-Python) captures automatically when your tool raises) and low-rated online-evaluator traces that used the tool. The judge proposes a rewritten definition - typically tightening parameter descriptions and enums against the failures it saw - shown as an editable diff, and nothing publishes until you approve it. Your agent's code then pulls the published definition instead of hardcoding it, the same source-of-truth pattern as the prompt registry.
The [Playground](/evaluation/playground) can also add a tool straight from this registry, so you can test a proposed definition against real models before publishing it.
The whole loop is scriptable via `client.evaluations.tool_schemas` - the tool-definition
analog of the [prompt registry's SDK path](/improve/prompt-management#the-same-loop-from-the-sdk):
```python theme={null}
schema = client.evaluations.tool_schemas.get_or_create(
name="search_orders",
description="Order lookup for the support agent",
definition=json.dumps(my_tool_definition),
)
evidence = client.evaluations.tool_schemas.examples(schema["_id"], window="7d")
proposal = client.evaluations.tool_schemas.propose(schema["_id"], window="7d")
if proposal.get("proposal"):
p = proposal["proposal"]
client.evaluations.tool_schemas.publish_version(
schema["_id"],
definition=p["definition"],
reasoning=p["reasoning"],
based_on_version=p["basedOnVersion"],
)
```
Before publishing a proposed definition, [validate it](/improve/validating-proposals): both versions run against the tool's own production failures (or a dataset), measured on tool choice and argument validity.
## Tools you haven't registered yet
Failure classification was never registry-gated: `agent-tool-failure:` events tally for any traced tool (an operational outcome, not a scorer - it feeds the KPI metrics and this evidence loop directly, without raising triage signals). What registration unlocks is the improvement loop - so the Tool Schemas view has a **Registered / Unregistered** toggle, with live counts on each side. **Unregistered** lists tool names observed in recent traces that aren't in the registry yet, with call/failure counts and a drafted definition ready to review - verbatim from the trace's `metadata.tools` when available, otherwise inferred from the arguments the model actually formed (marked as inferred).
The SDK's raw-client integrations (`patch_openai_client`, `patch_anthropic_client`, the LiteLLM logger, Google GenAI) and the LangChain/LangGraph handler **capture the request's `tools=[...]` definitions automatically** and attach them as `metadata.tools` - so any tool your agent actually passed to the model surfaces here with its real name, description, and parameter schema, ready to register unchanged. Manual tracing can do the same with `tracer.trace(..., metadata={"tools": my_tools_array})`. Click **Register**, review the draft, save - and because the failure history was keyed by the tool's name all along, the accumulated evidence feeds Suggest improvement immediately.
## Test endpoint
A registered tool can optionally carry a **test endpoint** - an http(s) URL the Playground's "From Tools & MCPs" picker uses as the tool's endpoint default, so a real-loop test of a registered tool needs no re-typing. Calls POST `{tool, arguments}` and expect `{result}` back. It is only ever called from a Playground run or a conversation simulation you start: the registry itself stays execution-free, and production traffic always runs your agent's own real tool. Set or clear it any time via `PATCH /evaluate/tool-schemas/:id/test-endpoint`.
## MCP tools
Tools served by a remote MCP server register into this same registry - see
[MCPs](/improve/mcp) for connecting servers (including OAuth-protected ones) and how
Playground runs execute MCP tools over the real protocol.
# Validating Proposals
Source: https://developers.agentx.so/improve/validating-proposals
Every proposed prompt or tool-schema rewrite ships with a measured verdict before a human publishes it
[Prompt Management](/improve/prompt-management) and [Tools](/improve/tool-schemas) generate improvement proposals from real evidence - but a plausible diff isn't proof. **Validate before publishing** runs the candidate and the current published version against the same real cases, under identical conditions, and shows the measured comparison. The human approving a rewrite approves a number ("candidate scored 7.8 vs 6.1 baseline on 18 cases, 2 regressions"), not a hunch.
## Prompts
In the proposal dialog, after generating a suggestion: pick a dataset and **Run validation**. Every case runs twice - once with the current published prompt as the system prompt, once with the candidate (including any edits you made in the diff) - graded with that dataset's own judge criteria. Multi-turn cases play out in full, each turn's real reply threaded into the next.
* The verdict compares **pairwise-scored** cases only (both variants got a rating), so one variant erroring can't skew the averages.
* Per-case deltas are listed, regressions highlighted - a candidate that wins on average but breaks two specific cases shows exactly which two.
* Publishing appends the verdict to the version's reasoning, so version history keeps the receipt.
## Tool schemas
Same flow, different measurement: both definitions are given to the model for the same real queries, and each run is scored on whether the model **chose the tool** and **formed valid arguments** against the definition's JSON schema: required arguments present, no unexpected keys, and each value checked against its property's `type`, `enum`, and string `pattern` - so a "digits only" schema actually catches `"#88231"`. Tool calls are simulated, never executed - what a tool *definition* controls is tool choice and argument shape, and that's exactly what's measured.
By default the queries come from the tool's own **production failure evidence** - the exact requests the current definition mishandled - or pick a dataset instead.
## An honest approximation
Validation runs dataset cases against (system prompt + a stock model), not your full agent - AgentX doesn't own your code, the same boundary [Model Portability](/evaluation/model-portability) documents. Baseline and candidate run under identical conditions, so the **delta** is the meaningful number even where absolute scores differ from your real agent's. For the highest-fidelity signal, validate against cases [curated from production](/evaluation/datasets-from-production): real inputs, real failure modes.
## From the API
```bash theme={null}
curl -X POST http://localhost:4700/api/v1/evaluate/prompts//proposals/validate \
-H "x-api-key: $KEY" -H "Content-Type: application/json" \
-d '{"candidateText": "You are a meticulous support agent...", "datasetId": ""}'
curl -X POST http://localhost:4700/api/v1/evaluate/tool-schemas//proposals/validate \
-H "x-api-key: $KEY" -H "Content-Type: application/json" \
-d '{"candidateDefinition": "{\"name\": \"search_orders\", ...}"}'
```
Both return the per-case pairs plus `{baselineAvg, candidateAvg, delta, regressions, verdict, summary}` (`verdict` is `improved` / `regressed` / `tie` / `insufficient`). Optional: `model` (default `gpt-4o-mini`) and `maxCases` (default 12; cases with expected answers are prioritized).
## The Improvement Inbox
Validation's fully automatic form: a background sweep (every 10 minutes) watches every registered prompt and tool schema, and when one accumulates **3+ pieces of fresh evidence in 24h** (recorded tool-call failures for a tool schema; examples rated below 5/10 for a prompt), it generates the proposal *and* runs the validation on its own, then queues the result at the top of **Insights → Suggestions** - trigger reason, verdict chip, reasoning, and the full diff, with **Publish** and **Dismiss** as the only actions. The human's job collapses to reading a measured verdict.
Spend and noise controls: at most 2 new proposals per sweep, a 24-hour per-target cooldown (dismissing means "stop nagging", not "retry in ten minutes"), never a second proposal while one is pending, and validation capped at 6 cases. Prompt validation uses the dataset from the most recent eval run tagged with that prompt's name; with no tagged run, the proposal queues unvalidated rather than not at all. `AGENTX_IMPROVEMENT_SWEEP=false` disables the background sweep; the **Check now** button (or `POST /evaluate/improve/inbox/sweep/run`) triggers one on demand.
Where prompt proposals come from
Where tool-definition proposals come from
# GitHub Actions
Source: https://developers.agentx.so/integrations/github-actions
Gate PRs with the hosted AgentX CI/CD evaluation, using GitHub Actions as the runner
Block merges and deploys when your agent's pass rate drops below your configured threshold,
using GitHub Actions as the runner. This page covers the **hosted** CI/CD pipeline
(`client.tracer.run_eval`); for a self-hosted engine, see the
[Self-Host CI Gate](/integrations/self-host-ci).
## Prerequisites
1. A dataset with **CI/CD enabled** in AgentX settings.
2. Your `AGENTX_API_KEY` stored as a GitHub Actions secret.
3. Your `AGENTX_DATASET_ID` (visible in the dataset settings page URL), stored as a secret or
variable.
## Workflow template
Use a small eval script plus a workflow that runs it. First the script:
```python theme={null}
# scripts/eval_gate.py
import os
import sys
from agentx import AgentX, CIGateFailure
from your_package import my_agent # import your agent: a function query -> answer string
client = AgentX.from_env()
try:
result = client.tracer.run_eval(
dataset_id=os.environ["AGENTX_DATASET_ID"],
agent_fn=my_agent,
agent_name="my-agent",
concurrency=4,
# Keys must be camelCase. The API's gitContext schema is strict
# and silently drops unrecognized (e.g. snake_case) keys.
git_context={
"branch": os.environ.get("GITHUB_HEAD_REF", ""),
"commitSha": os.environ.get("GITHUB_SHA", ""),
"prNumber": os.environ.get("GITHUB_REF", "").split("/")[-2],
"repoUrl": "https://github.com/" + os.environ.get("GITHUB_REPOSITORY", ""),
"triggeredBy": "github-actions",
},
fail_on_gate=True,
)
print(f"Gate PASSED, {result.pass_rate:.0%} ({result.passed_questions}/{result.total_questions})")
except CIGateFailure as e:
r = e.result
print(f"Gate FAILED, {r.pass_rate:.0%} ({r.passed_questions}/{r.total_questions})")
for v in r.violations:
print(f" Q{v.question_index}: {v.metric} = {v.actual:.2f} (threshold {v.threshold:.2f})")
sys.exit(1)
```
Then save this as `.github/workflows/eval.yml` in your repository:
```yaml theme={null}
name: AgentX Evaluation Gate
on:
pull_request:
branches: [main]
jobs:
eval:
name: Evaluate agent
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: pip install agentx-python
- name: Run AgentX evaluation gate
run: python scripts/eval_gate.py
env:
AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
AGENTX_DATASET_ID: ${{ secrets.AGENTX_DATASET_ID }}
```
`run_eval` runs the full lifecycle in one call: it creates a CI run, calls your agent for each
test case, submits each answer for judge scoring, finalizes the run, and returns (or raises on)
the gate decision. `fail_on_gate=True` raises `CIGateFailure` when the gate fails, so the step
exits non-zero and the check goes red.
Expected log output:
```
Gate PASSED, 88% (7/8)
```
## Adding required status checks
To **block merges** when the gate fails:
1. Go to your GitHub repo → **Settings → Branches**
2. Add a branch protection rule for `main`
3. Enable **Require status checks to pass before merging**
4. Add `eval / Evaluate agent` as a required check
## Viewing results
Every CI run is recorded in your AgentX workspace's evaluation run history, with:
* Gate result (PASS / FAIL)
* Per-question scores and justifications
* Branch, commit SHA, and PR number (from `git_context`)
* Threshold violations
You can also poll a run from code with `client.tracer.get_ci_run(run_id)`, or receive the
result by [webhook](/integrations/webhook) when the run finalizes.
## Environment variables
| Variable | Description |
| ------------------- | --------------------------------------------------------------------- |
| `AGENTX_API_KEY` | Your project API key (stored as a repository secret) |
| `AGENTX_DATASET_ID` | Evaluation dataset ID with CI enabled |
| `GITHUB_HEAD_REF` | PR branch name (set by GitHub Actions) |
| `GITHUB_SHA` | Commit SHA (set by GitHub Actions) |
| `GITHUB_REF` | Full ref string, `refs/pull//merge` on PRs (set by GitHub Actions) |
| `GITHUB_REPOSITORY` | `org/repo` (set by GitHub Actions) |
# Self-Host CI Gate
Source: https://developers.agentx.so/integrations/self-host-ci
Block a merge when your agent's eval score drops - an exit-code contract over a normal eval run
Run your golden dataset against the PR's version of your agent, then **gate** the run: an absolute rating floor, a no-regression check against the dataset's previous run, or both. The gate is a plain exit-code contract, so it drops into any CI system.
```python theme={null}
# ci/eval_gate.py
import sys
from agentx import AgentX
client = AgentX.from_env() # reads AGENTX_API_BASE_URL and AGENTX_API_KEY
def my_agent(case):
# your agent logic here
return answer_string
run = (
client.evaluations
.run(dataset_id="", subject={"kind": "custom_agent", "framework": "raw_python"})
.execute(my_agent) # the PR's version of your agent
.finalize()
)
gate = run.gate(fail_under=7, no_regression=True, caller="github-actions")
sys.exit(gate.exit_code) # 0 = merge, 1 = block
```
The two checks:
* **`fail_under`** - the run's average judge rating must be at or above the floor. Works from the very first run.
* **`no_regression`** - the average must not drop more than `tolerance` (default `0.5`) below the dataset's previous completed run. The tolerance exists because judge scores are noisy: an exact comparison would flake builds on variance, not regressions. Requires run history, so point CI at a **persistent** self-host instance rather than an ephemeral one.
`gate()` prints a per-check verdict into the CI log and returns a `GateResult` (`passed`, `exit_code`, `average_rating`, `baseline_average`, `checks`). The same check is a plain HTTP call if you'd rather script it raw - add `record=true` (what the SDK sends by default) so the verdict lands in the dashboard's gate history, with an optional `caller` label:
```bash theme={null}
curl "$AGENTX_API_BASE_URL/custom-agent-evaluations/runs//gate?failUnder=7&noRegression=true&record=true&caller=github-actions" \
-H "x-api-key: $AGENTX_API_KEY"
```
## GitHub Actions
Pointing at a persistent self-host instance (recommended - run history enables `no_regression`):
```yaml theme={null}
name: Eval gate
on: pull_request
jobs:
eval-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install agentx-python
- name: Run evals and gate
env:
AGENTX_API_BASE_URL: ${{ secrets.AGENTX_SELFHOST_URL }} # https://agentx.internal:4700/api/v1
AGENTX_API_KEY: ${{ secrets.AGENTX_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} # whatever your agent itself needs
run: python ci/eval_gate.py # the script above
```
No persistent instance reachable from CI? Launch an ephemeral engine inside the job and gate on the floor only (a fresh database has no baseline to regress against):
```yaml theme={null}
- name: Start ephemeral engine
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} # the engine's judge needs a provider key
run: |
pip install agentx-python
AGENTX_TRACE_EVAL_SKIP_WEB=1 nohup agentx-trace-eval > engine.log 2>&1 &
# First run downloads the engine release, so wait on /health rather than sleeping
timeout 120 bash -c 'until curl -fsS http://localhost:4700/health; do sleep 2; done'
- name: Run evals and gate
env:
AGENTX_API_BASE_URL: http://localhost:4700/api/v1
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} # whatever your agent itself needs
run: |
# In the default auth-disabled mode the engine hands out the Default project's key
export AGENTX_API_KEY=$(curl -fsS http://localhost:4700/api/v1/auth/config | python3 -c "import json,sys; print(json.load(sys.stdin)['apiKey'])")
python ci/eval_gate.py
```
## What to gate on
Every recorded gate lands in the dashboard's **CI Gates** history - the same list is
readable from the SDK for scripting (e.g. a weekly "how often did gates fire" report):
```python theme={null}
for g in client.evaluations.list_gates():
print(g["passed"], g["averageRating"], g["caller"], g["checks"])
```
The strongest gate is a dataset [curated from production](/evaluation/datasets-from-production): real failures as regression cases mean "the PR doesn't re-break what production already taught you." Pair `fail_under` (an absolute quality bar) with `no_regression` (this PR made nothing worse) - they catch different failure modes.
## Gate history in the dashboard
Every gate the SDK runs is **recorded by default** (`record=True`, with a `caller` label like `"github-actions"`), and the dashboard's **CI Gates** tab (in the sidebar under Automations) lists that history newest-first: verdict, dataset, average vs baseline, which checks ran, who called. The same page has a **preview** ("would the latest run pass these thresholds?") that is never recorded - so exploring thresholds can't pollute the history real CI jobs write - plus the copy-paste setup snippets. Pass `record=False` to `gate_run()` for an unrecorded check from code.
Grow the golden dataset the gate runs against
The hosted platform's equivalent pipeline
The same gate from a TypeScript pipeline, via @agentx/eval
# Webhooks
Source: https://developers.agentx.so/integrations/webhook
Receive CI gate results via HTTP callback
Configure a webhook URL on a dataset to have AgentX POST the gate result to your server immediately after a [hosted CI/CD run](/integrations/github-actions) is finalized. (The [self-host gate](/integrations/self-host-ci) returns its verdict synchronously instead - gate history is queryable, no webhook involved.)
## Configuration
Set `webhookUrl` on the dataset in **Settings → Datasets → \[dataset] → CI/CD**, or include it when creating a dataset via the API:
```json theme={null}
{
"ci": {
"enabled": true,
"webhookUrl": "https://your-server.com/hooks/agentx"
}
}
```
## Delivery
* Sent as a single `POST` request with `Content-Type: application/json`
* **Fire-and-forget**: AgentX does not retry on failure or timeout
* Delivery is best-effort; build idempotency into your handler using `run_id`
## Payload
```json theme={null}
{
"event": "ci_run.finalized",
"run_id": "6876ccc111aaa222bbb333dd",
"dataset_id": "6876ddd222bbb333ccc444ee",
"gate": "pass",
"pass_rate": 0.875,
"git_context": {
"branch": "feat/new-retrieval",
"commitSha": "a1b2c3d4e5f6",
"prNumber": 42,
"repoUrl": "https://github.com/org/repo",
"triggeredBy": "github-actions"
},
"scores": [
{
"question_index": 0,
"rating": 4,
"justification": "The agent correctly described the password reset flow.",
"passed": true,
"input": "How do I reset my password?",
"output": "Click Forgot Password on the login screen."
},
{
"question_index": 1,
"rating": 2,
"justification": "The agent gave an incorrect billing answer.",
"passed": false,
"input": "Why was I charged twice?",
"output": "..."
}
],
"finalized_at": "2026-07-06T11:00:00.000Z"
}
```
### Fields
| Field | Type | Description |
| -------------- | -------------------- | ---------------------------------------------------------- |
| `event` | string | Always `"ci_run.finalized"` |
| `run_id` | string | CI run ID |
| `dataset_id` | string | Dataset (EvaluationSettings) ID |
| `gate` | `"pass"` \| `"fail"` | Gate result |
| `pass_rate` | number | Fraction of questions that passed (0.0-1.0) |
| `git_context` | object \| null | Branch, commit SHA, PR number, etc. (from `create_ci_run`) |
| `scores` | array | Per-question scores (see below) |
| `finalized_at` | string | ISO 8601 finalization timestamp |
### `scores[n]` fields
| Field | Type | Description |
| ---------------- | -------------- | ----------------------------------------------------- |
| `question_index` | number | 0-based question index |
| `rating` | number | LLM score, 0-10 |
| `justification` | string | LLM explanation |
| `passed` | boolean | Whether the question passed all threshold gates |
| `input` | string \| null | Question text (null when `exposeTestInputs` is false) |
| `output` | string \| null | Agent's response text |
## Example receiver
```python theme={null}
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route("/hooks/agentx", methods=["POST"])
def handle_agentx_webhook():
payload = request.get_json()
if payload.get("event") != "ci_run.finalized":
return jsonify({"ok": True})
run_id = payload["run_id"]
gate = payload["gate"]
pass_rate = payload["pass_rate"]
branch = (payload.get("git_context") or {}).get("branch", "unknown")
print(f"[{branch}] CI run {run_id}: {gate.upper()} ({pass_rate:.0%})")
if gate == "fail":
# notify Slack, update GitHub status, etc.
notify_team(run_id, payload["scores"])
return jsonify({"ok": True}), 200
```
```typescript theme={null}
import express from "express";
const app = express();
app.use(express.json());
app.post("/hooks/agentx", (req, res) => {
const { event, run_id, gate, pass_rate, git_context, scores } = req.body;
if (event !== "ci_run.finalized") return res.json({ ok: true });
const branch = git_context?.branch ?? "unknown";
console.log(`[${branch}] CI run ${run_id}: ${gate.toUpperCase()} (${Math.round(pass_rate * 100)}%)`);
if (gate === "fail") {
const failing = scores.filter((s: any) => !s.passed);
console.log(`Failing questions: ${failing.map((s: any) => s.question_index).join(", ")}`);
}
res.json({ ok: true });
});
```
## Verifying delivery
Because AgentX does not sign webhook payloads currently, restrict your endpoint to known AgentX IP ranges or use the `run_id` to fetch the authoritative result from the API:
```python theme={null}
import requests
@app.route("/hooks/agentx", methods=["POST"])
def handle():
payload = request.get_json()
run_id = payload["run_id"]
# Verify with the API
result = requests.get(
f"https://api.agentx.so/api/v1/ingest/ci-runs/{run_id}",
headers={"x-api-key": API_KEY},
).json()
assert result["gate"] == payload["gate"]
return "", 200
```
# Introduction
Source: https://developers.agentx.so/introduction
Trace, monitor, evaluate, and improve AI agents - self-hosted, framework-agnostic, with online and offline evaluation
AgentX is a governance platform for AI agents. It records what your agents actually do in
production, judges that behavior continuously, scores changes against test datasets before they
ship, and turns everything it learns into concrete improvements - without touching your agent's
own code or prompts.
It is built for teams running AI agents in production (or about to): engineers who need to see
what an agent did, reviewers who need to judge whether it was right, and release owners who need
proof that a change made things better before it ships.
* **Framework-agnostic**: LangChain/LangGraph, OpenAI Agents SDK, CrewAI, AutoGen, Google ADK,
Google GenAI (Gemini), LlamaIndex, LiteLLM, direct OpenAI/Anthropic clients,
Databricks/MLflow, Moveworks, plain Python, or any OpenTelemetry-instrumented app.
* **Online and offline evaluation**: judges score live traffic as it arrives (traces, whole
conversations, tool behavior), and dataset runs score candidate changes before release.
* **Trajectory-aware**: evaluation sees the path the agent took - which tools, in what order,
with what failures - not just the final answer.
* **Self-hosted**: one local binary or container, bring your own LLM keys. SQLite by default,
Postgres for teams, Postgres + ClickHouse for enterprise telemetry volume. Your traces never
leave your infrastructure.
![PLACEHOLDER: circular diagram with four nodes - Trace (agent icon, arrows from LangChain/OpenAI/OTel logos), Monitor (judge/magnifier icon producing Signals), Evaluate (dataset + checkmark icon), Improve (wrench icon feeding back into the agent). Arrows flow clockwise: Trace to Monitor to Evaluate to Improve and back to the agent.]()
## Choose your path
Self-host the engine and send your first trace in five minutes
Decorators, context managers, sessions, and per-framework integrations
Dataset runs on demand: LLM-as-judge scoring, similarity metrics, trajectory matching
Continuous scoring of live traffic: patterns, LLM judge scorers, deduped signals
Fail the pipeline when quality drops below your bar
Prompt and tool registries with evidence-backed, validated proposals
## How it fits together
Four capabilities share one SDK, one trace store, and one dataset model. Each works alone;
together they form a loop.
| Capability | What it does | Where it lives |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Trace** | Records every agent interaction as a span tree - inputs, outputs, tool calls, latency, tokens, cost | Observe tab, [SDK tracing](/sdk/tracing) |
| **Online Evaluation** | Enabled scorers (patterns, LLM judge scorers, custom endpoint and code scorers) check live traffic continuously, deduped into reviewable signals | Scorers + Monitor tabs, [SDK monitor](/sdk/monitor) |
| **Offline Evaluation** | Scores your agent against versioned datasets on demand - judge ratings, similarity metrics, code scorers, expected-trajectory matching | Evaluate tab, [Evaluations SDK](/sdk/evaluations/overview) |
| **Improve** | Turns findings into registered prompt/tool changes, validated against datasets before you adopt them | Insights tab (Suggestions), Prompts and Tools & MCPs under the sidebar's Manage section, [Prompt Management](/improve/prompt-management) |
AgentX sits **outside** your agent in every case: tracing wraps your code with a decorator,
context manager, or framework callback; evaluation calls your agent function and scores what
comes back. There is no prompt injection and no change to your agent's logic.
## Deployment model
The self-hosted stack ([AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-Trace-Eval)) is a
single engine binary plus the real AgentX dashboard, installed via `curl | bash`, the Python
launcher (`pip install agentx-python`, then `agentx-trace-eval --dev`), or Docker. One binary,
three storage tiers: local SQLite by default, your Postgres with one env var, or Postgres +
ClickHouse for enterprise telemetry volume. You bring provider keys (OpenAI/Anthropic/Gemini)
only for the LLM-judge features - trace ingest and rule-based detection run with no keys at all.
Licensing: the engine and CLI are Elastic License 2.0 (use, modify, and run commercially;
just don't resell it as a hosted service). The [Python SDK](https://github.com/AgentX-ai/AgentX-Python)
is Apache-2.0, so nothing restrictive ever lives inside your application. The dashboard ships as
a prebuilt bundle with every release.
Everything documented on this site runs against a self-hosted instance. Start at
[Self-Host Overview](/self-host/overview).
# Coming from DeepEval
Source: https://developers.agentx.so/monitor/coming-from-deepeval
Metric-name mapping, what carries over, and what works differently
DeepEval is a metrics library; AgentX is a self-host evaluation platform. If you've built evals
on DeepEval, almost everything you wrote maps directly - and the platform half (live scoring,
review, calibration, dashboards) is what you gain. This page is the honest mapping, based on a
side-by-side run of identical test cases through both systems.
## Metric mapping
| DeepEval | AgentX | Notes |
| ------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FaithfulnessMetric` | **RAG: Faithfulness** | AgentX is stricter by design: claims must be *supported by* the context, not merely non-contradicting. In our comparison, a fully unsupported answer scored 1.00 on DeepEval ("no contradictions") and 1/10 here. |
| `AnswerRelevancyMetric` | **RAG: Answer Relevancy** | Both judge direction only, never correctness (that's Faithfulness's job). |
| `ContextualRelevancyMetric` | **RAG: Context Relevancy** | AgentX judges holistically with per-chunk reasoning; DeepEval scores a statement ratio, which can cliff (a retrieval containing the exact answer scored 0.00 in our comparison when a sibling chunk diluted the ratio). |
| `ContextualPrecisionMetric` | **RAG: Contextual Precision** | Same contract, including "no relevant chunks scores 0". |
| `ContextualRecallMetric` | **RAG: Contextual Recall** | Both require a reference answer. DeepEval raises `MissingTestCaseParamsError` mid-run without one; AgentX marks the scorer **Needs a reference answer** - live scoring is refused up front and reference-less dataset cases skip with a reason. |
| `GEval(criteria=...)` | **Any judge scorer** | Your GEval criteria paste straight into a scorer's acceptance/rejection/evaluation criteria - and the same scorer then also scores live traffic. |
| `TaskCompletionMetric` | **Agent: Task Completion** | Trajectory-anchored: the judge sees the tools called, in order, with failures. |
| `ToolCorrectnessMetric` | **Agent: Tool Correctness** + `expectedTools` on cases | For exact matching, dataset cases' `expected_tools` is deterministic - zero LLM cost. The judge template covers the semantic cases exact matching can't. |
| `StepEfficiencyMetric` | **Agent: Step Efficiency** | Loops, redundant calls, dead ends - judged from the trajectory. |
| `KnowledgeRetentionMetric` | **Session: Knowledge Retention** | Enable live scoring with per-session scope; the judge grades the whole transcript. |
| `RoleAdherenceMetric` | **Session: Role Adherence** | Same session mechanism. |
| `ToxicityMetric` / `BiasMetric` | **Safety: Harmful Content** / **Safety: Bias & Fairness** | Judge-based, meaning-level. The zero-cost template patterns (secrets, PII, profanity) catch literal matches without any LLM spend. |
| `assert_test` (pytest) | `agentx.testing.assert_evaluation` | Same ergonomic; the verdict is also recorded in the dashboard's CI gate history. |
## What works differently (on purpose)
* **One scorer, both surfaces.** A DeepEval metric is an offline construct; online evaluation
lives in its Confident AI cloud. An AgentX judge scorer grades dataset runs *and* live traffic
with one rubric - see [the two grading modes](/monitor/scorers#the-two-grading-modes).
* **0-10 with prose, not 0-1 ratios.** DeepEval scores are claim/statement ratios (auditable
arithmetic, occasional cliffs). AgentX judges holistically and states its counts in the
justification ("2 of 3 claims supported") - auditable prose, no cliffs.
* **Hallucinations fail hard.** DeepEval's ratio math gave a known contradiction 0.50 in our
comparison; AgentX's exact-fact rule floors a wrong core fact to 0-3 regardless of polish.
## What we don't have (and where to go instead)
* **Academic benchmarks** (MMLU, HumanEval, GSM8K...): DeepEval ships 17. These benchmark
foundation models, not your agents - if you need them, run DeepEval's benchmark suite; it
coexists fine with AgentX in the same codebase.
* **Dataset synthesizer**: AgentX builds datasets [from production traces](/evaluation/datasets-from-production)
and generates grounded cases from pasted source text (plain text or Markdown only); DeepEval's
synthesizer adds document-file parsing and larger-scale generation.
* **Red-teaming**: use their `deepteam`; AgentX's safety scorers monitor production, they don't attack it.
* **Multimodal metrics** (image, audio, voice): DeepEval only.
## Migration in practice
```python theme={null}
# DeepEval
metric = GEval(name="Helpfulness", criteria="Answers using the retrieved policy...")
assert_test(LLMTestCase(input=q, actual_output=a), [metric])
# AgentX - same rubric, plus it now also scores live traffic
scorer = client.monitor.judge_scorers.create(
name="Helpfulness",
judge={"acceptanceCriteria": "Answers using the retrieved policy..."},
)
report = client.evaluations.run(dataset_id=ds, scorer_id=scorer.id).execute(agent).finalize()
assert_evaluation(report, min_rating=7.0, no_regression=True)
```
# External & Code Scorers
Source: https://developers.agentx.so/monitor/custom-evaluators
Your own scoring logic - as an HTTP endpoint, or a script run in-engine
For checks that need your own logic - a proprietary classifier, a business rule that needs data
outside the trace, anything code can decide that a phrase/regex/semantic pattern can't - two
scorer kinds run YOUR code against a sample of live traffic:
* An **external scorer** POSTs each sampled trace to your own HTTP endpoint and uses its verdict.
* A **code scorer** runs a script you write (Python or JavaScript) inside the engine itself - no
endpoint to host.
Like [LLM judge scorers](/monitor/online-evaluators), both are standalone entities on the
[Scorers page](/monitor/scorers) with their own sample rate, agent scope, enable/pause toggle,
and severity. (The SDK/API resource name for both remains `custom-evaluators`.)
## Code scorers
**Scorers → New scorer → Code scorer.** The script defines one function:
```python theme={null}
from typing import Any
# handler returns a numeric score between 0 and 1,
# or a dict with 'score' and optional 'metadata' and 'name' fields,
# or None to skip scoring this trace.
async def handler(
input: Any,
output: Any,
expected: Any, # always None for live traffic
metadata: dict[str, Any],
trace: Any,
) -> float | dict[str, Any] | None:
all_spans = await trace.get_spans()
llm_spans = await trace.get_spans(span_type=['llm'])
return {
'name': 'span count scorer',
'score': 1.0 if output else 0.0,
'metadata': {
'total_span_count': len(all_spans),
'llm_span_count': len(llm_spans),
},
}
```
The JavaScript variant is identical in shape (`async function handler(...)`,
`await trace.getSpans({ spanType: ["llm"] })`). A returned score **below the scorer's alert
threshold** (default 0.5) raises a signal; every scored check is recorded in the scorer's event
history either way, and `None`/`null` skips the trace entirely.
`trace.get_spans()` returns the trace's own span subtree (root first, start-time ordered). Each
span carries `span_id`, `parent_span_id`, `name`, `input`, `output`, `error`, `model`,
`latency_ms`, `input_tokens`, `output_tokens`, `tool_calls`, `metadata`, `started_at`, and a
derived `type`. Child spans classify as `llm` (a model is recorded), `tool` (tool calls
recorded), `retrieval` (`metadata.kind == "retrieval"`), or `span` (anything else); the root
span instead carries the trace's resolved [span kind](/trace/span-kinds) (`agent`, `chain`, ...),
so match the root by its `parent_span_id` being null rather than by `type == "span"`.
| Behavior | Detail |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Runtime | JavaScript: in-process sandbox. Python: a `python3` subprocess (present in the Docker image; on bare installs the scorer reports clearly if `python3` is missing) |
| Budget | 8 seconds per trace, per scorer |
| Failures | A crashing/timing-out script records an errored check in the scorer's history and never raises a signal or breaks ingest |
| Security | Not a sandbox boundary: a scorer is code the operator chose to run on their own engine. Don't hand scorer creation to anyone who shouldn't run code on the machine |
**Dry run** in the dialog executes the script against a synthetic two-span sample (one root, one
`llm` span) and shows the score it returns.
## External scorers
**Request** (engine → your endpoint, `POST`, JSON, `schemaVersion: 2`):
```json theme={null}
{
"schemaVersion": 2,
"evaluatorId": "abc123",
"evaluatorName": "Policy checker",
"agentId": "vN2k...",
"traceId": "trace-id",
"trace": {
"input": "...", "output": "...", "error": null, "toolCalls": [],
"name": "support-agent", "model": "gpt-4o-mini", "framework": "openai",
"sessionId": "sess-1", "spanId": "root-span",
"latencyMs": 1450, "inputTokens": 220, "outputTokens": 64,
"cacheReadTokens": null, "cacheWriteTokens": null,
"metadata": { "channel": "chat" },
"startedAt": "2026-08-22T01:02:03.000Z", "createdAt": "2026-08-22T01:02:04.000Z"
},
"spans": [
{ "span_id": "root-span", "parent_span_id": null, "name": "support-agent", "type": "span", "input": "...", "output": "...", "error": null, "model": null, "latency_ms": 1450, "input_tokens": null, "output_tokens": null, "tool_calls": null, "metadata": null, "started_at": "..." },
{ "span_id": "llm-1", "parent_span_id": "root-span", "name": "LLM Call 1", "type": "llm", "model": "gpt-4o-mini", "latency_ms": 900, "input_tokens": 220, "output_tokens": 64, "input": "...", "output": "...", "error": null, "tool_calls": null, "metadata": null, "started_at": "..." }
]
}
```
Everything the tracer records rides along: the full root record under `trace` (the v1 keys -
`input`/`output`/`error`/`toolCalls` - are exactly where they always were, so v1 endpoints keep
working untouched), and the trace's span subtree under `spans` with the same per-span fields and
derived `type` values code scorers see (`llm`/`tool`/`retrieval`/`span`).
Two things endpoint authors trip on: `agentId` is AgentX's **internal agent id**, never the
agent's name (fetch the agent list once rather than string-comparing names). And
`evaluatorId`/`agentId`/`traceId` are all **`null` in a dry run** - the first payload your
endpoint will ever receive - so handle nulls before assuming strings.
**Response your endpoint must return:**
```json theme={null}
{ "matches": true, "reason": "optional, shown on the resulting signal", "score": 7.5 }
```
`matches` is the only field that decides anything. `reason` and `score` are both optional and purely informational - recorded and shown alongside the resulting signal, but neither affects whether one is raised. A minimal example endpoint (Flask):
```python theme={null}
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.post("/evaluate")
def evaluate():
payload = request.json
output = payload["trace"]["output"] or ""
matched = "cannot help" in output.lower()
return jsonify({"matches": matched, "reason": "contains a refusal phrase" if matched else None})
```
## Setup and behavior
Build it from the dashboard: **Scorers → New scorer → External scorer**. The dialog
documents the request/response contract in place, and **Dry run** sends a synthetic v2 payload to
your URL so you can confirm it responds correctly before saving.
| Behavior | Detail |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Raise when | Your choice of `matches: true` or `false` raising the signal (defaults to `true`), with a severity like any evaluator |
| Timeout | Fixed 8 seconds |
| Failures | An erroring/timing-out/malformed response skips that one check for that trace (logged, not raised); every other evaluator still runs |
| Auth | None built in - embed a token in the URL's query string, the same way a Slack incoming-webhook URL does |
Distinct from `monitor_profiles.channels`' `"webhook:"` notification targets - those fire-and-forget a message *after* a signal is raised and never consume a response. A custom evaluator's response *is* the detection result, awaited synchronously (up to the 8-second timeout).
The Python SDK covers the whole lifecycle as `client.monitor.scorers`: `list()`, `create_code(...)`, `create_external(...)`, `update(id, ...)`, `delete(id)`, `events(id)`, and `dry_run(...)`. The REST equivalents live on `/agent-monitoring/custom-evaluators`: `GET`/`POST` on the collection, `PUT`/`DELETE` on `/:id`, `GET /:id/events` for the check history (code scorers pass `{"kind": "code", "language": "python"|"javascript", "script": "...", "alertBelow": 0.5}`), plus `POST /agent-monitoring/custom-evaluators/dry-run` (transient - `{"url": ...}` for external, `{"kind": "code", "language": ..., "script": ...}` for code - the live response back out, nothing persisted).
# Model Comparison
Source: https://developers.agentx.so/monitor/model-comparison
Per-model health, failure rate, latency, and cost aggregated from real traffic
Distinct from [Model portability](/evaluation/model-portability) (a what-if replay of one trace): Overview's **Model comparison** card aggregates what your traffic already shows - every model observed in real traces over the window, with trace count, health rate, failure rate, p95 latency, and estimated cost side by side. Health and failure come from Monitor's own event ledger, cost from measured token usage priced against the [model catalog](/evaluation/model-portability#the-pricing-catalog). No LLM calls involved, it's a pure read of data you already have; a metric with no data shows "-" rather than a fake 100% or \$0.00. Use it to notice that the cheaper model your team switched half the traffic to is also failing twice as often.
# LLM Judge Scorers
Source: https://developers.agentx.so/monitor/online-evaluators
A real LLM judge scoring live traffic continuously, per trace or per session
Monitor's built-in and custom patterns check traces against rules (does the output contain X, does it match a rubric); online evaluators are different - a real LLM judge scores a sample of your live traffic continuously, the same judge-scoring logic Evaluate's offline runs use, just pointed at production instead of a golden dataset. This is what most people mean by "online evals": catching quality drift on real traffic, not just outright failures.
Since the judge-scorer unification, an LLM Judge Scorer is ONE entity with three parts: the **judge rubric** (acceptance/rejection/evaluation criteria, judge prompt, judge model), an **offline eval config** (how dataset runs grade with it - repetitions, similarity metrics, code scorers), and an **online eval config** (how it scores live traffic - the "online evaluator" this page describes). Any judge scorer can grade both surfaces. Manage everything from the [Scorers page](/monitor/scorers) (kind: **LLM judge**): one editor writes all of it atomically, and every save through the editor upserts the online config row - disabled by default, so live scoring is gated by the online config's **Score live traffic** switch (off = the config stays saved, nothing is scored live, no LLM spend). On the wire, `online: null` means the scorer has no online profile yet - an SDK create that omitted `online`, a seeded template, or a template you just added. A scorer in that state has no live-traffic evidence, so its row menu hides **Ratings** and shows **Tune judge** disabled with a tooltip telling you to enable online scoring first; flipping its row toggle opens the editor rather than silently going live - going live is always an explicit save. Engine-seeded quick-start templates (the RAG/Agent/Session/Safety metric packs and the Example Helpfulness Judge) carry `seeded: true` on the wire; the Scorers page shows its first-run starter state until a non-seeded scorer exists, and lists templates under **New scorer → Template scorer** (Add clones the template into your own scorer) instead of padding the table. The table sorts newest first. Create one via **New scorer** → **LLM judge scorer**; pause its live scoring with the row's toggle, and open **Ratings** for a rating-over-time chart.
The editor is a three-step modal (per the claude.design Judge Scorer Modal):
1. **Judge** - the rubric (acceptance / rejection / guidance; acceptance criteria is
*recommended, not required* - offline runs grade against the dataset's expected results, so
the default template alone is a valid judge, while live and session scoring lean on your
criteria), with **Judge model & prompt** as a collapsible machinery card beneath it (model,
tool-context level, template - auto-expanded when a custom template exists). The right rail
is **Try it on real traffic**: pick any recent trace *or a whole multi-turn session* and run
ONE judge call with the unsaved draft (`GET /agent-monitoring/judge-scorers/preview-context`
lists the targets, `POST .../preview-score` scores; sessions go through the sweep's exact
structured session prompt), followed by live assembly cards showing exactly what the judge
receives.
2. **Live scoring** - the Score live traffic switch, scope (per trace / per session + idle
seconds), sample rate, signal threshold and severity, agent restriction, and a
judge-calls/day estimate from your last 7 days of traffic.
3. **Offline runs** - repetitions, deterministic metrics, code scorers, default-for-runs.
Saving requires only a name.
Strictly one online config row per scorer. The legacy SDK/API surfaces (`online_evaluators` with an `evaluation_settings_id` reference, and `evaluation-settings`) keep working as views onto the same entity - the SDK clients now emit a (default-hidden) DeprecationWarning pointing at `client.monitor.judge_scorers`, and the dashboard no longer uses them anywhere - binding a config that is already another evaluator's profile transparently binds a fresh copy instead of sharing.
A score below the evaluator's **alert threshold** (0-10, default 5) raises a signal, the same triage surface a failing Monitor pattern already lands on, deduped by evaluator and agent so a recurring low score accumulates one occurrence count instead of a new signal per trace. Set the threshold's **severity** (low/medium/high/critical), or turn the threshold off entirely to score purely for the ratings chart with no triage. A signal raised this way shows the evaluator's own name and an "LLM judge" tag in the Signals list, distinguishing it from a pattern match at a glance; clicking through opens that evaluator's ratings dialog directly.
The same thing works from the Python SDK:
```python theme={null}
# The unified surface (preferred): rubric + profiles in one call.
scorer = client.monitor.judge_scorers.create(
"Helpfulness",
judge={"acceptanceCriteria": "Actually resolves the user's question."},
online={"enabled": True, "sampleRate": 0.1, "alertThreshold": 5, "severity": "medium"},
)
client.monitor.judge_scorers.ratings(scorer.id, window="7d")
client.evaluations.run(dataset_id, subject, scorer_id=scorer.id) # same rubric offline
# Or the snake_case builder - rubric, offline config, and live profile in one call:
scorer = client.monitor.judge_scorers.builder(
"Helpfulness", acceptance_criteria="Actually resolves the user's question.",
live=True, sample_rate=0.1, alert_threshold=5,
).publish()
# The legacy per-profile surface still works unchanged:
evaluator = client.monitor.online_evaluators.builder(
name="Helpfulness",
evaluation_settings_id=settings.id,
sample_rate=0.1,
alert_threshold=5,
severity="medium",
).publish()
client.monitor.online_evaluators.get(evaluator.id)
client.monitor.online_evaluators.list()
client.monitor.online_evaluators.update(evaluator.id, alert_threshold=None) # score only, never raise a signal
client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
client.monitor.online_evaluators.events(evaluator.id, window="7d") # individually scored traces behind a ratings point
client.monitor.online_evaluators.delete(evaluator.id)
```
`sample_rate` (default `0.1`) matters more here than it does for pattern-matching: every check is a real LLM call against your own API key. `scope_mode`/`agent_ids` restrict an evaluator to specific agents, same as a pattern's. Results land in the same event log Overview's KPI/trend widgets read from, but are excluded from that health-rate math (they're a continuous score, not a failure signal). For a hard spend ceiling on top of sampling, set `AGENTX_QUOTA_ONLINE_JUDGE_CALLS_PER_DAY`: a daily per-project cap on live trace-scope judge calls, unset = unlimited (the broader `AGENTX_QUOTA_JUDGE_CALLS_PER_DAY` cap - per organization in multi-tenant mode, per instance otherwise - still applies separately).
### When the judge fails
A judge call that returns nothing usable (empty response, unparseable verdict) is retried once
automatically. If the retry also fails, the failure is never disguised as a score:
* **Offline runs**: the result is stored with status `"skipped"` and a `null` rating - excluded
from `averageRating` and from [CI gates](/integrations/self-host-ci), never averaged in as a
zero. The run's `liveStatistics` carries `skippedCount` and `failedCount` so a run that
silently judged nothing is visible at a glance.
* **Online scoring**: no Signal is raised and no rating is recorded. An
`online_eval_judge_failure` event is kept instead, and the
[Scorers page](/monitor/scorers) shows a per-scorer judge-failure count - a judge that has
quietly stopped scoring looks broken there, not healthy.
### What the judge actually receives
The editable template carries four substitutable variables - `{input}`, `{output}`,
`{expected}`, and `{context}` (retrieval context from `metadata.retrievalContext` or recorded
retrieval spans) - but the judge always sees more than the template. The engine assembles the
final message in a fixed order, and a custom prompt can never silently lose the extra context:
1. **Your template**, with the four variables substituted.
2. **Agent execution trajectory** - tools called with arguments and results, order, failures,
rendered from the trace's span tree (when a trace is linked). Governed by the scorer's
**Tool context** level (`judge.toolContext`): `"none"` strips this block entirely (the
judge sees only conversation + expected results - for pure text rubrics), `"simple"` (the
default, the historical behavior) includes it, `"detailed"` adds the block below.
3. **Tool definitions** (`toolContext: "detailed"` only) - full definitions of each tool the
agent actually *used*, deduped: captured `metadata.tools` from the trace's spans first
(the exact schema the model saw - the SDK integrations record every request's `tools=[...]`
list), with the Tools & MCPs registry as a by-name fallback for manual tracer users. Plus
one line naming tools advertised to the model but not used, so "was there a better tool it
didn't pick?" stays gradeable without paying for unused schemas. The used-names-only
registry lookup is what keeps the project-wide registry safe - it can never inject another
agent's tools.
4. **Judge Guideline** - the dataset case's per-question guideline (offline runs only).
5. **Acceptance Criteria**, **Rejection Criteria**, and **Evaluation Criteria** (the editor's
"Additional guidance" field) - appended verbatim as labeled blocks.
Two special cases: when a case has no expected results, the default template swaps to a
reference-free variant that judges against your criteria alone (a custom template gets a
substitution that defuses its expected-results rules in place); and per-session scoring skips
the template entirely, building a structured whole-transcript prompt from your criteria -
filtered to the same Tool context level (`"none"` = conversation turns only, `"detailed"` adds
the used-tool definitions to the transcript). The
editor's "What the judge receives" cards render this order live as you edit, with expanders for
the exact assembled prompt and a fully filled example.
### Trajectory-aware judging
Per-trace scoring is trajectory-aware: the judge prompt includes the agent's actual execution
path - tools called with their arguments and results, order, failures - rendered from the
trace's span tree (or its flat tool-call list). Criteria written against the trajectory
("no unnecessary or repeated tool calls", "answers must follow from tool outputs") are
scoreable, not just aspirational - and the **Detailed** tool-context level adds the used tools'
full definitions so "was there a better tool it didn't pick?" is scoreable too.
### Per-trace vs. per-session scope
By default an evaluator scores individual traces as they arrive (`scope="trace"`). Pass `scope="session"` to judge **whole conversations** instead: a background sweep watches for multi-turn sessions that have been quiet for `idle_seconds` (default 120), builds the full transcript, and scores it against the same evaluator criteria - so criteria like "did the conversation actually resolve the customer's problem" become answerable. See [Multi-Turn Session Evaluation](/monitor/session-evaluation) for the full guide: sweep mechanics, the built-in Session Baseline Judge, on-demand SDK checks, and turning failing conversations into regression tests.
Every project ships with one built-in session evaluator: **Session Baseline Judge**, created paused like every other scorer - switch it on from the Scorers page. It scores whole-conversation consistency - context retention, self-contradiction, goal drift - and points at the first step where the conversation broke. Its rubric is a normal judge rubric (not code), so **Tune judge** improves it from calibration evidence like any other scorer; the scorer itself is read-only except the rubric, the prompt, and the live-scoring switch, and can't be deleted.
```python theme={null}
client.monitor.judge_scorers.builder(
"Conversation resolution",
acceptance_criteria="The customer's problem is actually resolved by the end.",
live=True,
scope="session",
idle_seconds=120,
alert_threshold=5,
).publish()
```
The dashboard's evaluator editor has the same Scope card (Per trace / Per session). `AGENTX_SESSION_SWEEP=false` disables the background sweep; `POST /agent-monitoring/session-sweep/run` triggers one manually either way.
Full lifecycle from the SDK - `list`/`get`/`update`/`delete` alongside the builder, plus the
scored evidence behind each evaluator:
```python theme={null}
evaluators = client.monitor.online_evaluators.list()
client.monitor.online_evaluators.update(evaluator_id, sample_rate=0.25) # or enabled=False to pause
ratings = client.monitor.online_evaluators.ratings(evaluator_id, window="7d") # bucketed averages
events = client.monitor.online_evaluators.events(evaluator_id, window="7d") # worst-rated traces + justifications
```
## Tuning the judge from the SDK
The dashboard's **Tune judge** flow (measure the evaluator against recorded reality, rewrite
its criteria from the disagreements, validate by exact re-judging, publish behind approval) is
fully scriptable:
```python theme={null}
cal = client.monitor.online_evaluators.calibration(evaluator_id, window="rubric")
# cal["agreements"], cal["missed"], cal["overFlagged"], cal["disagreementCases"]
# cal["alpha"] - chance-corrected agreement (see below); cal["ratingMae"] - mean absolute
# error against human re-scores, over the pairs that carry a corrected number
proposal = client.monitor.online_evaluators.tune(evaluator_id, window="7d")
criteria = {k: proposal[k] for k in ("acceptanceCriteria", "rejectionCriteria", "evaluationCriteria")}
verdict = client.monitor.online_evaluators.validate_tuning(evaluator_id, criteria, window="7d")
client.monitor.online_evaluators.publish_tuning(evaluator_id, criteria, validation=verdict)
```
The `window` argument accepts `"24h"`, `"7d"`, `"30d"`, or `"rubric"`. `"rubric"` - the
dashboard's Tune Judge default - means *verdicts produced by the current rubric*: everything
since the scorer's criteria were last edited or a tuning proposal was published, clamped to 30
days. That is the semantically right evidence set for tuning, because disagreements older than
the last rewrite are complaints about criteria that no longer exist, and tuning from them
re-litigates issues a previous rewrite already fixed. The time windows remain useful for
scorers with sparse review coverage, where the current rubric hasn't accumulated ground truth
yet. The response's `window` and `since` fields echo the boundary that was actually applied.
### Reading the agreement numbers
`agreementRate` is raw agreement, and raw agreement flatters a judge on imbalanced traffic: if
90% of responses are genuinely fine, a judge that passes everything scores 90% while detecting
nothing. `alpha` is the same comparison corrected for chance - Krippendorff's alpha over the
binary verdict pair, which for two raters is Scott's pi with a small-sample correction. Read it
as: 1 = perfect alignment, 0 = no better than a coin weighted like the traffic, negative =
systematically opposed. `alphaBand` ships a plain-language band (poor / slight / fair /
moderate / substantial / near-perfect) so every surface interprets the same value the same way.
Two honesty rules. First, `alpha` is `null` until `alphaMinItems` verdicts have ground truth -
with a handful of labels the statistic swings wildly, so it is withheld rather than fabricated.
Second, the ground-truth streams (triage corrections, review labels, disputes) skew toward
contested cases by construction, so read `alpha` as *alignment on human-reviewed items*, not on
raw traffic - a genuinely fine judge can look mediocre if humans only ever reviewed the hard
calls. Sampled labels from the [review queue](/monitor/review-queue) reduce that skew.
Ground truth comes from [outcomes, user feedback](/monitor/outcomes), and
[human review-queue labels](/monitor/review-queue); the validation verdict
is measured (candidate criteria re-judge the exact cases the current ones got wrong, plus a
control set they got right), not estimated. Publishing is gated on that provenance: pass the
`validate_tuning` result as `validation`, and a `"regressed"` verdict is refused - `force=True`
is the deliberate escape hatch, recorded as such in the version history.
## Built-in metric pack
Every project ships with seeded judge-scorer templates (`seeded: true` on the wire) - twelve across four families (RAG, Agent, Session, Safety; the full list is on the [Scorers page](/monitor/scorers)). They are pickable as the grader anywhere one is chosen (dataset runs, the Playground) and live in **New scorer → Template scorer** rather than padding the Scorers table - **Add** clones one into your own editable, deletable scorer. The RAG five:
* **RAG: Faithfulness** - every factual claim in the response must be supported by the retrieved context.
* **RAG: Answer Relevancy** - does the response actually address the query?
* **RAG: Context Relevancy** - judges the *retriever*: were the retrieved chunks relevant and sufficient? A low score means fix retrieval, not the prompt.
* **RAG: Contextual Precision** - are the relevant chunks ranked above the irrelevant ones? Targets the reranker.
* **RAG: Contextual Recall** - does the context cover what the expected answer needs? Offline-oriented (needs a case's `expected_results`).
The RAG prompts reference `{context}`, which resolves from a trace's `metadata.retrievalContext` (a string or an array of chunk strings) when set, else from the trace's own recorded **retrieval spans** (`tracer.trace_retrieval(...)`, LangChain/LlamaIndex retriever callbacks, OTel retrieval spans) - so if your integration already records retrievals, the judges see the chunks with no caller changes. See [RAG evaluation](/evaluation/rag) for the full metric guide and the offline context sources. Add one from the template picker to get an ordinary scorer you can edit, tune with **Tune judge**, or delete - the template itself stays in the library. A zero-cost **PII in response** built-in pattern (emails, phone numbers, SSNs, payment cards, regex-based) also runs alongside the other built-in checks.
# Outcomes & Judge Calibration
Source: https://developers.agentx.so/monitor/outcomes
Report what actually happened, and measure the judges against it
Patterns, [online evaluators](/monitor/online-evaluators), and eval runs all score your agent with LLM judges - but did the judge get it right? Self-host lets you report what **actually happened** after the fact, then measures the judges against it:
```python theme={null}
client.outcomes.report(
trace_id=trace_id,
outcome="reopened", # a free label in your own taxonomy
is_negative=True, # the polarity calibration compares against
reason="Customer reopened the ticket within 3 days",
reported_by="servicenow-webhook",
)
```
The intended caller is usually another system - an incident tracker's webhook, a CRM workflow - via `POST /outcomes` (`{"traceId", "outcome", "isNegative", "reason?", "reportedBy?"}`; `evaluationRunResultId` works in place of `traceId` for offline eval results). Overview's **Judge calibration** card compares each report against whatever verdict AgentX recorded for the same trace at the time - pattern hits, online-evaluator scores - and shows the agreement rate: how often a negative outcome was flagged in advance (and how often a good one was wrongly flagged). That turns "trust the LLM judge" into a measured number, and tells you which evaluators are worth tightening.
Per-evaluator calibration is also readable from the SDK - agreement rate, misses,
over-flags, and the exact disagreement cases - and feeds straight into the scriptable
[judge-tuning loop](/monitor/online-evaluators#tuning-the-judge-from-the-sdk):
```python theme={null}
cal = client.monitor.online_evaluators.calibration(evaluator_id, window="7d")
print(f"{cal['agreements']}/{cal['withGroundTruth']} agreement, missed {cal['missed']}")
print(f"chance-corrected: {cal['alpha']}") # Krippendorff's alpha; null until enough labels
```
The project-level roll-up (the Judge calibration card's own numbers - compared count,
agreement, false-positive and false-negative rates across ALL of AgentX's verdicts, plus the
chance-corrected `alpha` explained on the
[online evaluators page](/monitor/online-evaluators#reading-the-agreement-numbers)) is one
call:
```python theme={null}
cal = client.monitor.calibration(window="7d")
print(f"compared={cal['comparedCount']} agreement={cal['agreementRate']:.2f} "
f"FP={cal['falsePositiveRate']:.2f} FN={cal['falseNegativeRate']:.2f}")
```
## End-user feedback
The other ground-truth stream is your own users' votes, forwarded with
`client.feedback.report(...)` - every vote (up and down) feeds the same calibration math as
outcome reports, so the judges get measured against real human reactions too. See
[User Feedback](/monitor/user-feedback) for the full story (trace chips, the triage signal,
the Downvote rate KPI).
Both streams are deliberately kept out of the "AgentX flagged it in advance" side of
calibration: they are the reports being calibrated against, never predictions that inflate
agreement.
## Human labels on unflagged traffic
Both streams above still depend on something reaching you. Neither can catch a judge quietly
scoring bad answers as good, because those traces never surface. The
[human review queue](/monitor/review-queue) is the third stream and the one that closes that
gap: traces that raised no signal at all, labeled good or bad by a person, optionally with a
corrected score next to what the judge gave. Those labels feed this same confusion matrix,
counted separately as `reviewLabelCount`.
## Tuning the judges themselves
Calibration doesn't just measure - it feeds back. Each online evaluator has its own agreement rate against recorded reality, split by direction: **missed** (judge passed it, reality said bad - criteria too generous) and **over-flagged** (judge flagged it, reality said fine - criteria too strict). From the evaluator's row menu, **Tune judge** opens the disagreement cases - when several ground-truth sources cover the same case, a human re-score from signal triage wins, since it carries a rationale - and generates a rewrite of the evaluator's own grading criteria that encodes the principles behind them.
The disagreement evidence draws on all three ground-truth streams: outcome reports, user
feedback, and [review-queue labels](/monitor/review-queue) - a human "bad" with a corrected
score next to the judge's 8.5 is exactly the kind of case a rewrite gets built from.
What makes this loop stronger than the prompt/tool ones: judging is exactly reproducible, so validation isn't an approximation. The candidate criteria **re-judge the very cases the current criteria got wrong**, plus a control set of cases they got right (the anti-overfit guard), and agreement with recorded ground truth is measured directly: "agrees on 3/5 cases the current criteria got wrong, preserves 4/5 they got right (net +2)". Publishing updates the evaluator's config with normal version history, so every tuning is reversible.
Publishing is also **gated on that verdict**: the publish call must carry the validation result
(the SDK's `publish_tuning(..., validation=verdict)`, or `validation` in the request body), and
a measured regression is refused outright. `force: true` publishes anyway - a deliberate
escape hatch, not a default. The published version's history entry records the provenance
either way: `[judge tuning: validated improved, net agreement +2]` for a validated publish,
`[judge tuning: published without validation]` for a forced one - so six months later you can
still tell which criteria changes were measured and which were vibes.
# Patterns
Source: https://developers.agentx.so/monitor/patterns
Rule-based failure detection - phrase, regex, and semantic conditions that turn matching traces into signals
A **pattern** is a detection rule checked against your traffic: when a trace matches, a **signal** is raised (or a healthy tally recorded) in the same triage queue everything else feeds. Patterns are the deterministic kind on the [Scorers page](/monitor/scorers) - cheap and exact - alongside the six shipped built-in templates, while [LLM judge scorers](/monitor/online-evaluators) are the judgment kind, scoring quality continuously with a real LLM. Use a pattern when you can say precisely what bad looks like ("promises a refund", "mentions a competitor", "apologizes more than once"); use an LLM judge when you can only describe it.
## Building a pattern
**Scorers** → **New scorer** → **Pattern scorer**. A pattern is one or more **condition rows**, evaluated top to bottom:
* **Detector** - each row is one of three kinds:
* **Phrase**: plain-text contains match (case sensitivity is a per-row toggle).
* **Regex**: a regular expression body. Don't want to write one? The row's **Generate with AI** popover drafts the regex from a plain-English description with an LLM call, right in the dialog - review and test it before saving.
* **Semantic**: a rubric an LLM judges the text against ("The response promises a refund."). The only detector kind that needs a provider key configured.
* **Match target** - where the row looks: `response` (the traced output), `userMessage` (the traced input), or `trace` (output plus error plus every recorded tool call's name/input/output, flattened - the right target for "did any tool mention X").
* **Negate** - flips a single row's verdict before it joins the others.
* **Connector** - rows combine top to bottom with **AND**, **OR**, or **NOR** (joins as "and not"), so "contains 'refund' AND NOT matches the approved-refund-template regex" is two rows.
## Behavior settings
| Setting | Effect |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Signal level** | One slider, `Info` / `Low` / `Medium` / `High` / `Critical`. The four upper levels raise a signal to triage with that severity; `Info` (wire: `polarity: "proper"`) records the match without raising a signal - informational, counted as neither a failure nor a healthy run |
| **Sample rate** | Fraction of eligible traces actually checked - the only sampling applied (there is no global monitoring rate multiplying in). Matters most for semantic rows, where every check is a real LLM call |
| **Enabled** | The dialog's Enforcement switch, mirrored by the Scorers list's inline toggle - off means the pattern never runs |
Agent scoping (`scopeMode` / `agentIds` - all agents, or only the ones you name) exists on the
wire and is enforced at detection time, but the pattern dialog doesn't expose a picker for it
yet - set it via the API if you need it.
Phrase and regex detection work with **no provider keys at all** - only semantic rows need one.
## Where matches go
Detection stops at the first matching pattern per trace - one signal per trace - and matches are deduped by pattern and agent: a recurring issue accumulates an occurrence count on one signal instead of a new row per trace. From a signal's row on the **Review** tab you can record a verdict (**Confirm**, **Fixed**, **Ignore**, **Wrong judgement**), open the matched trace with **View trace**, or **Add to dataset** - turning the failure into a regression case, with an LLM assist that drafts the human feedback and expected results for you to edit.
Finished with a signal? **Archive** it (per row, or "Archive selected" in bulk) - archived
signals leave every filter except **Review → All signals**' explicit **Archived** filter, so the
queue never grows unbounded. Archiving is a shelf, not a grave: if the same issue fires again,
the signal reopens into the active list automatically.
## From the SDK
Patterns are first-class SDK resources: `client.monitor.patterns.builder(...).publish()` returns an id you can pass at trace time via `pattern_ids` - a trace sent with `pattern_ids` is checked against only those patterns (built-in checks and sampling are bypassed for it). The [Monitor](/sdk/monitor) page documents the builder's full parameter table, the built-in checks that run alongside your custom patterns, and reading signals back with `client.monitor.signals`.
The judgment half: continuous LLM scoring of live traffic
Delegate the verdict to your own HTTP endpoint
# Human Review Queue
Source: https://developers.agentx.so/monitor/review-queue
Label ordinary traffic, not just what got flagged, and measure the judge against it
Every judge calibration story starts the same way: you look at what the judges flagged, agree or
disagree, and tune. That loop has a hole in it. Reviewing only flagged traces can tell you the
judge cries wolf too often, but it can never tell you the judge is **quietly scoring bad answers
as good** - those traces never reach you, because nothing flagged them.
The review queue closes that hole. It holds traces that raised no signal at all, waiting for a
human verdict.
## Getting traces into it
Two ways, both visible in **Review → Sampled**:
Open any trace in Observe and click **Send to review**. Use it when something looks off and
you want a second opinion on the record.
A [rule](/monitor/rules) with the **Send to review** action samples live traffic into the
queue continuously - 10% of everything, or only errored traces, or only traces mentioning
"refund".
The queue is capped at 200 pending items. Past that, new traces are refused with a reason rather
than silently dropped, because a queue nobody can finish is not a backlog, it is a leak. Work it
down or narrow the rule.
## Labeling
The verdict vocabulary here is deliberately different from the signals queue. There is no claim
to confirm or reject - nothing was flagged - only a judgment on the answer itself:
* **Good** or **Bad**, plus an optional note.
* When a judge already scored that trace, a **corrected score** box appears: what the rating
should have been, 0-10.
That last pair is the point. A human "bad" next to a judge's 8.5 is exactly the evidence that a
judge is too generous, and it is the pair
[outcome calibration](/monitor/outcomes) consumes. The score box only appears when there is a
score to correct, because a corrected rating with nothing to compare it against is noise.
`j` / `k` move through the queue, `g` labels good, `b` labels bad.
## What it feeds
Labels flow into the same confusion matrix as production outcome reports, reported separately as
`reviewLabelCount`, so you can see judge agreement measured on ordinary traffic rather than only
on the traffic the judge itself chose to flag. A judge that looks precise on flagged traces and
disagrees constantly on sampled ones is a judge with a blind spot, and this is the only place
that shows up.
## From the Python SDK
The whole loop is scriptable as `client.monitor.review_queue` - queue a suspicious trace from
the same script that found it, or batch-label from a notebook:
```python theme={null}
item = client.monitor.review_queue.queue(trace_id, note="Looks off, second opinion please")
for item in client.monitor.review_queue.list(status="pending"):
print(item.trace_id, item.judge_score_at_queue)
client.monitor.review_queue.label(item.id, "bad", corrected_score=2, note="Invented a refund window")
client.monitor.review_queue.dismiss(item.id) # remove without a verdict
```
`label` takes `"good"` or `"bad"`; `corrected_score` (0-10) only makes sense when a judge
already scored the trace. Labels and corrected scores feed per-scorer
[calibration](/monitor/outcomes) and become evidence for **Tune judge**, exactly like labels
recorded in the dashboard.
## API
```bash theme={null}
# Queue a trace
curl -X POST "$AGENTX_URL/api/v1/agent-monitoring/review-queue" \
-H "x-api-key: $AGENTX_API_KEY" -H "content-type: application/json" \
-d '{"traceId": ""}'
# What is waiting
curl "$AGENTX_URL/api/v1/agent-monitoring/review-queue?status=pending" \
-H "x-api-key: $AGENTX_API_KEY"
# Record a verdict
curl -X PATCH "$AGENTX_URL/api/v1/agent-monitoring/review-queue/" \
-H "x-api-key: $AGENTX_API_KEY" -H "content-type: application/json" \
-d '{"label": "bad", "correctedScore": 3, "note": "Invented a refund window"}'
```
Refusals are explicit and carry their reason: an unknown trace is a 404, a trace already waiting
is a 409, and a full queue is a 429. None of them is a silent success.
Labeled items are stored and included in [backups](/self-host/backup) - a labeled corpus is real
human work, and losing it on a restore would mean doing it all again.
# Automation Rules
Source: https://developers.agentx.so/monitor/rules
Route matching traffic to human review, into a dataset, or out to a webhook
A rule watches incoming traces and routes the ones that match somewhere useful. Three things it
can do, one per rule:
| Action | What lands where |
| ------------------- | ------------------------------------------------------------------------------------------- |
| **Send to review** | The trace joins the [human review queue](/monitor/review-queue) for a good/bad label. |
| **Add to dataset** | The trace becomes a dataset case, with the expected result left blank for a human to write. |
| **Post to webhook** | A small JSON payload (Slack-compatible) with the rule name and trace id. |
Rules live under **Manage → Rules**.
## Rules route, scorers score
This is the distinction worth getting right before you build one.
A **scorer** scores traffic, and owns its own sampling, because what a judge costs is a scorer
question. A **rule** routes traffic, and never scores anything.
The consequence people care about: **enabling a rule cannot change what your judges cost or what
verdicts they produce.** Routing is cheap - no LLM call - so the only reason to sample below 100%
of whatever matches the filter is queue volume, not the bill. (The dashboard starts a new rule at
10% for exactly that reason; a rule created via the API with no `sampleRate` runs at 100%.) Dial
the rule down when the queue fills up.
## Building one
A rule is a filter, a sample rate, and an action.
The filter has typed fields rather than an expression language:
* **Any trace** or **errored only**
* **Contains** - text that must appear in the input or output
* **Model** - exact match
(An agent scope - all agents, or a chosen few - also exists on the wire as `scopeMode`/`agentIds`
and is enforced at run time, but the rule editor doesn't expose a picker for it yet.)
This is a deliberate limit. A rule whose filter cannot match anything is visibly wrong in the
editor, instead of parsing cleanly into "match nothing" and looking healthy while doing nothing.
The sample rate then applies to whatever survived the filter: `10%` of matching traces, not 10%
of all traffic.
## Honest activity
Every rule shows either `fired 24x · last 3 minutes ago` or, plainly, **never fired**. A rule that
has never matched anything looks different from one working hard, because the most common failure
here is not a broken rule - it is a rule that quietly matches nothing while everyone assumes
coverage exists.
## Two rules worth having
10% of everything → **Send to review**. This is what keeps judge calibration measured on
normal traffic, not only on what got flagged.
Errored only → **Add to dataset**. Real production failures become regression cases. The
expected answer stays blank on purpose: what the agent said is what happened, not what
should have happened.
## API
```bash theme={null}
curl -X POST "$AGENTX_URL/api/v1/agent-monitoring/rules" \
-H "x-api-key: $AGENTX_API_KEY" -H "content-type: application/json" \
-d '{
"name": "Sample support traffic",
"action": "review",
"sampleRate": 0.1,
"filter": {"status": "any", "contains": "order"}
}'
```
An action missing its configuration is refused at creation - `dataset` without a `datasetId`,
`webhook` without a `url` - rather than stored as a rule that can never do anything. A rule action
that fails at run time is logged and isolated, so one broken rule never stops the others and never
fails an ingest.
Rules are included in [backups](/self-host/backup).
# Scorer Groups
Source: https://developers.agentx.so/monitor/scorer-groups
Compose scorers of any kind into one 0-10 score with weights and must-pass gates
A **scorer group** composes several scorers - LLM judges, patterns, code and external scorers -
into **one 0-10 score**. Members are stored by reference, so a group reuses the scorers your
project already has: editing a member scorer changes every group that includes it. The group
itself is then what a dataset run or live traffic is graded with, which answers the question
per-scorer configuration never could: *"I have five opinions about this response - what is THE
score?"*
Create and manage groups on the **Scorers** page's **Groups** sub-tab.
## How the score is computed
Every member keeps its native contract and is normalized to a 0-1 "goodness" before weighting:
| Member kind | Native scale | Normalized goodness |
| ---------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| LLM judge | 0-10 rating | rating ÷ 10 |
| Pattern (templates included) | matched / not matched | polarity-mapped: a matched **failure** pattern is 0, clean is 1; a matched **proper** pattern ("contains disclaimer") is 1 |
| Code scorer | 0..1 | as-is |
| External scorer | 0..1 score, or a boolean verdict | score as-is; a boolean maps 0/1 through the scorer's invert setting |
On the wire a member is `{ "kind", "refId", "weight", "gate" }`, with exactly three `kind`
values: `judge` (`refId` is an LLM judge scorer id), `pattern` (a pattern id), and `custom`
(a custom-evaluator id - this one kind covers both code and external scorers, since either
is a custom evaluator under the hood).
The group score is the **weighted average of member goodness, reported on 0-10** - so it reads
like a rating everywhere ratings already live: the run rating column, CI gates
(`gate(fail_under=...)`), and live alert thresholds. Two safeguards:
* **Renormalization**: weights average over the members that actually produced a score - a
deleted scorer or a judge outage shrinks the denominator instead of dragging the score down
(the failed member is still reported on the result, never hidden). A group whose every member
has weight 0 produces no score at all; gates still apply.
* **Must-pass gates**: a member marked *must pass* zeroes the whole group score when its
goodness lands below 0.5 (a matched failure pattern, a judge under 5/10, a code score under
0.5) - "every quality bar must pass", regardless of how good the blend looks.
A judge member contributes its **rubric verdict only**: its similarity metrics and [attached
code checks](/evaluation/code-scorers) belong to that scorer's own grading recipe and do not
run inside a group. Blending metrics into a score stays an attached-code-check job, scoped to
the scorer that owns those metrics.
## Grading a dataset run with a group
Pass the group's id as `scorer_group_id` (instead of `scorer_id`) - the group's aggregate fills
the rating column, judge members surface as labeled per-row verdicts, and deterministic members
as scorer rows:
```python theme={null}
group = client.monitor.scorer_groups.create(
"Release quality bar",
members=[
{"kind": "judge", "refId": quality.id, "weight": 1, "gate": False},
{"kind": "judge", "refId": safety.id, "weight": 1, "gate": True}, # must pass
{"kind": "pattern", "refId": pii.id, "weight": 0, "gate": True}, # gate only
],
)
run = client.evaluations.run(
dataset_id=dataset.id,
subject={"kind": "custom_agent", "displayName": "my-agent"},
scorer_group_id=group.id,
).execute(my_agent).finalize()
run.gate(fail_under=7) # gates on the GROUP score, like any rating
run.gate(fail_under=8, scorer="Safety") # or gate a NAMED MEMBER's own average
```
`scorer_group_id` and `scorer_id` are mutually exclusive - the group wins when both are sent.
Member verdicts land per row (judges in `judge_scorer_results`, deterministic members as
scorer rows), the run's breakdown leads with "*Name* (group)", and the Evaluate list's Scorers
column shows the group name. A member deleted after the group was built degrades to "not
scored" on future results rather than failing the run.
In the dashboard, the Create evaluation dialog offers your groups next to the judge scorer
picker ("Or grade with a scorer group"), and the generated snippet emits `scorer_group_id=`.
The **Playground**'s Scorers panel lists groups too - star one and it grades every grid cell
(the aggregate in the rating pill, member verdicts as chips); starred groups are saved with
Playground workbench profiles.
## Scoring live traffic
Give a group an `online` profile and it scores live traffic exactly the way an online
evaluator does: a group score below the alert threshold raises a Signal, events key on
`scorer-group:`, and the score history feeds the group's ratings chart. The profile's
`scope` picks the unit of judgment - `"trace"` (the default) scores each sampled trace as it
arrives, `"session"` scores whole multi-turn conversations once they go quiet.
### Scoring each trace
The default scope - every sampled ingested trace runs every member:
```python theme={null}
client.monitor.scorer_groups.update(group.id, online={
"enabled": True,
"sampleRate": 0.2, # 20% of traffic
"alertThreshold": 6, # Signal when the GROUP score lands below 6/10
"severity": "high", # scope defaults to "trace"
})
client.monitor.scorer_groups.ratings(group.id, window="7d") # score history for the chart
```
Live scoring has full parity with online evaluators:
* **Score chip**: when a group has scored a trace, the group score **owns the Score chip** in
Live Traces (the composed final opinion beats any individual verdict for the headline);
individual verdicts stay in the expanded row.
* **Trace Details**: a group-scored trace shows a **Group score / Judge scores** switch - the
group aggregate with its blend breakdown on one side, per-member verdicts (recorded
individually at scoring time) on the other.
* **Webhooks**: a below-threshold group score pages the agent's alert channels exactly like a
matched pattern or a low online-evaluator verdict.
* **Judge budget**: judge members draw from the same online judge-call budget online evaluators
reserve from - a group with 3 judge members takes 3 slots per sampled trace, and an exhausted
budget skips the group rather than scoring it on a partial panel.
* **KPIs**: group scores (and member verdicts) are ratings, not runs - they never inflate the
Overview's run counts.
Judge members are real LLM spend at the sample rate - the sample dial is the cost dial.
### Scoring whole sessions
Some failures don't live in any single reply - the agent contradicts turn 1 on turn 3, or the
conversation ends without the user getting what they came for. Set `scope: "session"` and the
group judges **the whole conversation as one unit**, on the same idle trigger
[session evaluation](/monitor/session-evaluation) uses: once a session with 2+ turns has been
quiet for `idleSeconds` (default 120), the background sweep scores it - and re-scores it
automatically if the conversation later grows. Individual traces of a session-scoped group are
deliberately **not** scored at ingest, so a conversation is never double-judged.
```python theme={null}
client.monitor.scorer_groups.update(group.id, online={
"enabled": True,
"sampleRate": 1.0, # of eligible idle sessions
"scope": "session",
"idleSeconds": 120, # judge once quiet for 2 minutes
"alertThreshold": 6, # Signal when the conversation's GROUP score lands below 6/10
"severity": "high",
})
```
Every member reads the same assembled transcript:
* **Judge members** are asked the structured whole-session question built from their criteria
(consistency across turns, whether the user's need was resolved). A judge's per-trace
`judgePrompt` never applies at session scope - the same rule session evaluators follow.
* **Pattern and code/external members** receive the full transcript as their content, so a
"contains" condition or a script scorer reads the conversation, not one turn.
* Weights, renormalization, and must-pass gates work identically - an apology tripwire pattern
with `weight: 0, gate: true` zeroes a conversation that apologizes anywhere.
The verdict lands in the session's judge rail (Governance > Observe > Sessions, labeled "*Name*
(group)"), joins the group's ratings history, and a below-threshold score raises a Signal
keyed on the same `scorer-group:` - and pages the agent's webhook channels exactly like a
below-threshold trace score. Judge members draw from the same online judge-call budget as
trace scoring - one slot per judge call actually made. `sampleRate` is a true per-conversation
fraction: the keep-or-skip decision is made once per (conversation, group) and stays stable
across sweep ticks, so 0.2 really means about a fifth of conversations get judged.
Reading verdicts back from the SDK:
```python theme={null}
client.monitor.sessions.scores(session_id)
# -> [{"kind": "scorer-group:", "rating": 4.5, "justification": "Weighted blend: ..."}, ...]
```
## SDK surface
`client.monitor.scorer_groups`: `list()`, `get(id)`, `create(name, members, description=None,
online=None)`, `update(id, **fields)` (sparse; `online=None` detaches live scoring),
`delete(id)`, and `ratings(id, window="7d")`.
The runnable, assertion-style walkthroughs are `sample-scripts/eval_deep_dive/09_scorer_groups.py` (dataset runs, gates, trace-scope live scoring) and `10_session_group_scoring.py` (session scope end to end).
Groups replace the older habit of chaining scorers inside one judge's [attached code
checks](/evaluation/code-scorers): those still work (and can still blend via `scores`), but a
group is reusable, visible in the catalog, and composes every scorer kind - not just code.
# Scorers
Source: https://developers.agentx.so/monitor/scorers
One catalog for everything that can score your traffic - nothing runs until you enable it
A **scorer** is anything that can pass judgment on your agent's traffic - and the same scorer
serves both evaluation surfaces: it can grade **live traffic** ([online evaluation](/sdk/monitor))
and **dataset runs** ([offline evaluation](/sdk/evaluations/overview)). That is why Scorers is its
own docs category rather than a sub-page of either one. The **Scorers** page is the single
catalog for all of them - one list, filtered by intent rather than split by implementation, with
an inline enable toggle per row. Three kinds live side by side:
| Kind | What it is | Cost |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| **Template / Pattern** | Deterministic rules - the six [shipped templates](/sdk/monitor) (secrets, PII, injection echo, profanity, refusal, malformed JSON) plus [patterns you author](/monitor/patterns) (phrase, regex, or LLM-judged rubric rows) | Zero LLM cost (except semantic rubric rows) |
| **LLM judge** | [One unified judge scorer](/monitor/online-evaluators): a rubric (criteria/prompt/model) plus an offline eval config (dataset-run grading) and an online eval config (how it scores live traffic - per trace, or per session once a conversation goes idle). Any judge scorer can grade online or offline; with live scoring off it scores nothing and costs nothing. Seeded judge templates live in **New scorer → Template scorer**, not the table - Add clones one into your own editable scorer | One judge call per sampled trace/session (live scoring only) |
| **Custom** | [Your own logic](/monitor/custom-evaluators): an external HTTP endpoint returning a verdict, or a code scorer - a Python/JavaScript `handler()` you write, run in-engine | Whatever your code does |
Everything is **opt-in**: a fresh project applies no scorer at all, and the page's footer says so
plainly - nothing is judged until a scorer is enabled. Operational outcomes (trace errors, tool
failures, empty responses) are *not* scorers and need no enabling: the trace itself recorded
them, and they feed the KPI failure metrics directly.
## In this section
* [Scorer groups](/monitor/scorer-groups) - compose any of the below into one weighted 0-10 score
* [LLM judge scorers](/monitor/online-evaluators) - one rubric, graded online and offline
* [Template & pattern scorers](/monitor/patterns) - deterministic rules, zero LLM cost
* [Custom scorers](/monitor/custom-evaluators) - your own endpoint or in-engine `handler()` code
* [Code scorers in offline runs](/evaluation/code-scorers) - the same code scorers attached to dataset grading
* [User feedback](/monitor/user-feedback) and [outcome reports](/monitor/outcomes) - the ground truth that calibrates your judges
* [Coming from DeepEval](/monitor/coming-from-deepeval) - metric-name mapping and an honest gap list
The seeded template catalog (New scorer -> Template scorer) ships four families: **RAG** (Faithfulness, Answer Relevancy, Context Relevancy, Contextual Precision, Contextual Recall), **Agent** (Task Completion, Tool Correctness, Step Efficiency - trajectory-anchored), **Session** (Knowledge Retention, Role Adherence - enable with per-session scope), and **Safety** (Harmful Content, Bias & Fairness - meaning-level judges next to the zero-cost content patterns).
## The two grading modes
The same scorer runs on two surfaces with one structural difference: **every offline dataset case
carries `expected_results`; no live trace ever does.** The judge handles this with one rubric and
two modes:
| | Online (live traffic) | Offline (dataset runs) |
| ---------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Rubric | Your acceptance / rejection / evaluation criteria | The same criteria |
| Reference answer | None exists - **reference-free** grading | The case's `expected_results` acts as **ground truth for facts**: claims that contradict it score low |
| Mode name | Reference-free | Reference-guided |
Because both modes anchor on the criteria, a 7 online and a 7 offline answer the *same question*
("does this meet the criteria?") - the reference just adds factual strictness where one exists.
This is the industry "reference-guided grading" pattern, applied so one scorer honestly serves
both surfaces instead of you maintaining two.
**The exception**: a rubric that is *about* the reference itself - RAG: Contextual Recall
("does the context cover what the expected answer needs?") or a custom "matches the reference
exactly" prompt - has nothing to grade without one. Mark it **Needs a reference answer** in the
editor: the engine then refuses to enable live scoring for it (HTTP 409) and offline runs skip
any case that lacks `expected_results` with an explicit reason, instead of judging emptiness.
## The list
* **Stat strip**: how many scorers are enabled, how many are firing this week, total signal
occurrences in the window, and how many are available but off.
* **Filters**: scope pills (All / Enabled / Firing / Off), a kind filter, and search. Rows sort
newest first.
* **The table is yours**: it lists what you created or enabled. Pre-existing material - built-in
patterns, seeded judge templates, the built-in session judge - stays in the **Template scorer**
picker until enabled or added, so a new project's table holds exactly what you made.
* **Signals · 7d**: the payoff column - each enabled scorer's signal count for the week with a
daily sparkline, "None yet" for enabled-but-quiet, "Not running" for disabled.
* **Enabled**: a toggle, right on the row. Built-in templates store this project-wide
(`enabledBuiltinPatterns` in monitoring defaults); the other kinds carry their own flag.
* **Row actions**: click a row (or its menu) to open the scorer's editor - built-in templates
open read-only with an "Exactly what it checks" list of their literal rules; LLM judges also
get **Ratings** (score-over-time) and **Tune judge** (rewrite criteria from calibration
evidence). Both are live-traffic surfaces, so a judge with no online profile yet hides
Ratings and shows Tune judge disabled, with a tooltip telling you to enable online scoring
and let it score some traffic first.
**New scorer** offers the four authorable kinds - a pattern (deterministic rules), an LLM judge
(criteria in plain language + a judge model), a code scorer (your own Python/JavaScript
`handler()`, run in-engine), or an external endpoint - plus **Template scorer**: the library of
ready-made material. Adding an LLM judge template clones it into your own editable copy;
built-in patterns and the session judge switch on in place.
## First run
For a clean project - nothing enabled and nothing of your own created yet - the page opens as a
starting point instead of a wall of paused rows: recommended templates to switch on with one
click (all zero-LLM-cost, plus the built-in Session Baseline Judge), or jump straight into
authoring your own. Creating any scorer of your own exits this state immediately, even with
live scoring off.
## Where scorer verdicts land
An enabled scorer's hits raise **signals** - deduped rows queued for verdicts on the **Review** tab, with [Monitor](/sdk/monitor) charting their volume and each live judge's score trend.
Judge scorers additionally record a rating per sampled trace whether or not it crossed the
alert threshold, feeding [Judge Calibration](/monitor/outcomes) against outcomes and
[user feedback](/monitor/user-feedback).
### What moves which metric
Not everything that raises a signal reclassifies the run - build alerting on the right surface:
| Event | Raises a triage signal | Counts into failureRate / run outcomes |
| --------------------------------------------------------------- | ---------------------- | ------------------------------------------------------------- |
| Operational outcome (trace error, tool failure, empty response) | No | **Yes** (operational failures) |
| Template/pattern scorer hit | Yes | **Yes** (scorer detections) |
| LLM judge score below threshold | Yes | No - judge scores are evaluator events, tracked per evaluator |
| Code / external scorer hit | Yes | No - same evaluator-event semantics as judge scores |
| User downvote | Yes | No - moves the Downvote rate KPI instead |
The practical consequence: an alert built on `failureRate` will see operational failures and
pattern hits, but NOT code-scorer, external-scorer, or judge verdicts - watch those scorers'
own signals and per-scorer event history (or the Scorers page's Signals column) instead.
## Scorers score, rules route
A neighboring concept worth keeping straight: an [automation rule](/monitor/rules) also watches
live traffic and also has a sample rate, but it never scores anything. It routes matching traces
into human review, into a dataset, or out to a webhook. Scoring costs a judge call and belongs to
a scorer; routing is free and belongs to a rule. Enabling a rule therefore cannot change what
your judges cost or what they decide.
# Multi-Turn Session Evaluation
Source: https://developers.agentx.so/monitor/session-evaluation
Judge whole conversations, not just single responses - automatically, once the conversation goes quiet
Per-trace scoring answers "was this response good?". Plenty of conversational failures are
invisible at that granularity: every individual reply looks fine, but the agent contradicts
what it said three turns ago, asks for the order number twice, drifts off the customer's goal,
or never actually resolves anything. **Session evaluation** judges the assembled multi-turn
conversation as one unit, so criteria like *"did the conversation resolve the customer's
problem"* become scoreable.
Any traces sharing a `session_id` form a [session](/trace/sessions) - pass it to the tracer,
or let framework integrations set it:
```python theme={null}
with client.tracer.trace("support-agent", session_id="conv-812", input=user_msg) as span:
...
```
## When does a conversation get judged?
Conversations have no end event - the engine can't know a user won't come back. Session
evaluation answers this the way trace/group-scoped online scoring tools converged on: **judge
once the session has been quiet**. A background sweep (every 60s) finds sessions idle longer
than the evaluator's `idle_seconds` (default 120) and judges the full transcript. If the
conversation later resumes, the next sweep re-judges it automatically - a session whose latest
score is newer than its last activity is never judged twice.
Bounds that keep this safe to leave on:
* Only sessions with **2+ turns** qualify - single-trace sessions are already covered by
per-trace scoring, and judging them again at conversation prices would double the bill.
* At most **5 sessions per sweep tick** are judged (a budget shared across all projects on the
instance, round-robin); anything left over is picked up next tick.
* The sweep looks back **24h** for candidates by default. `AGENTX_SESSION_SWEEP_WINDOW_HOURS`
widens that (values are bucketed: up to 24 means 24h, up to 168 means 7d, above that 30d) -
useful after downtime, or for traffic that arrives in delayed batches.
* `AGENTX_SESSION_SWEEP=false` disables the sweep entirely.
## The built-in: Session Baseline Judge
Every project ships with one built-in session evaluator: the **Session Baseline Judge**. Like
every scorer it is created paused - switch it on from the [Scorers page](/monitor/scorers). It
scores whole-conversation consistency - context retention, self-contradiction, goal drift,
non-repetition - and points at the first span where the conversation broke. Its rubric is an
ordinary judge-scorer rubric, so **Tune judge** improves it from
[calibration evidence](/monitor/outcomes) like any other evaluator; the row itself can be
paused but not deleted.
Alongside the 0-10 score, every session judgment returns structured **findings**: up to six
per-step citations, each with a short category tag, rendered as the judge rail in the
session's detail view with the cited turns flagged inline.
## Your own session criteria
Create a session-scoped evaluator exactly like a per-trace one - the only difference is
`scope="session"`:
```python theme={null}
client.monitor.judge_scorers.builder(
"Conversation resolution",
acceptance_criteria=(
"The conversation ends with the customer's problem resolved or a concrete, "
"dated next step. No information the customer already gave is asked for again."
),
rejection_criteria=(
"The agent contradicts an earlier turn, loops on clarifying questions, "
"or the customer gives up."
),
live=True,
scope="session",
idle_seconds=120,
alert_threshold=5, # a failing conversation raises a signal
severity="high",
).publish()
```
A verdict below `alert_threshold` raises a signal in the normal
[triage queue](/monitor/patterns), deduped like every other signal source. Verdicts also
appear on the session's detail view and in the **Session score** column of Observe →
Sessions.
Need more than one opinion per conversation? A [scorer group](/monitor/scorer-groups) can be
session-scoped too (`online: {"scope": "session", ...}`): several judges, patterns, and code
scorers read the same transcript and blend into one 0-10 verdict on the same idle trigger,
with must-pass gates - e.g. an apology tripwire that zeroes the conversation's score.
## On demand from the SDK
The sweep is the automatic path; both checks also run on demand - the same judging, same
score shape, and an explicit call ignores the evaluator's paused state (a human asking for a
score should get one):
```python theme={null}
score = client.monitor.sessions.coherence_check(session_id) # one judge call
print(f"{score['rating']}/10 across {score['spanCount']} spans")
print(score["justification"])
if score["driftSpanId"]:
# The drift span can be any child in the session - walk it up to its turn root.
spans = client.monitor.sessions.spans(session_id)
by_id = {s.get("spanId"): s for s in spans if s.get("spanId")}
drift = next((s for s in spans if s["_id"] == score["driftSpanId"]), None)
while drift and drift.get("parentSpanId"):
drift = by_id.get(drift["parentSpanId"])
```
Two more helpers round out the surface:
```python theme={null}
# Every session-level verdict, newest first. `kind` says who scored: "online-eval:"
# for a session evaluator, "scorer-group:" for a scorer group, or legacy "coherence"
# rows from before the Session Baseline Judge existed.
for s in client.monitor.sessions.scores(session_id):
print(s["kind"], s["rating"], s["justification"])
# Run the idle-session sweep once, right now, over THIS project's idle sessions (it runs
# automatically every minute) - useful in demos, tests, and backfills where waiting a tick
# is the wrong UX. Concurrent calls are serialized: a sweep already in flight returns
# {"judged": 0, "skipped": true} instead of double-judging the same sessions.
client.monitor.sessions.run_sweep() # -> {"judged": n}
```
## Closing the loop
A failing conversation is the best kind of regression test - multi-turn, real, and specific.
**Add to dataset** from the session view converts the whole conversation into a dataset case
(follow-up turns included, provenance kept), so the fix gets guarded by
[offline runs](/sdk/evaluations/overview) and [CI gates](/sdk/ci-cd). Simulated conversations
from the [Playground](/evaluation/simulate-conversation) are recorded as real sessions too,
so the same judging and the same loop apply to rehearsals before any real user is involved.
The full runnable flow - weak prompt, scripted multi-turn session, per-turn and session-level
judging, then prompt/tool improvement proposals from the evidence - is
[`selfhost_demo/10_session_coherence_and_tool_improvement.py`](https://github.com/AgentX-ai/AgentX-Sample-Scripts/blob/main/selfhost_demo/10_session_coherence_and_tool_improvement.py).
# Topics
Source: https://developers.agentx.so/monitor/topics
What your agents are actually being asked, clustered
![PLACEHOLDER: entity-relationship style diagram. Left: Agent emits Traces (with child Spans and Tool calls), Traces group into Sessions. Middle: Patterns and Online Evaluators read Traces/Sessions and raise Signals. Right: Datasets (Questions + criteria) drive Evaluation Runs producing Results with ratings. Bottom: Prompts and Tool Schemas registries receive evidence from Signals and Results.]()
Monitor tells you when your agent fails; **Topics** tells you what people actually ask it. When enabled, an LLM classifier assigns each monitored trace a short topic label (plus sentiment and issue type, and an embedding used for clustering - the embedding step needs `OPENAI_API_KEY`), clustered and shown as Overview's **Topic map** card and the dashboard's own **Topics** tab - the fastest way to see that 40% of traffic is about one thing your prompt barely covers. It's off by default (every classification is an LLM call against your key): turn it on with the **Topics classification** switch at the top of the **Topics** page itself - a single project-level toggle, applied instantly - and pick the classification rate next to it ("Classifies X% of traffic"), Topics' own sampling knob for keeping judge spend proportional to volume. Classification is per trace, not per session, since a topic describes an individual request.
# User Feedback
Source: https://developers.agentx.so/monitor/user-feedback
Forward your users' thumbs up/down votes - the cheapest ground truth there is
Your app already renders the agent's answers, and your users already know which ones were bad.
A vote button next to each response, forwarded to AgentX, gives you human ground truth with zero
LLM cost: it triages real complaints, powers the **Downvote rate** KPI, and calibrates every
automated judge against actual human reactions.
Feedback is deliberately **not a scorer**. Scorers (the Scorers tab - template patterns, LLM
judges, custom endpoints) are judgments you opt into; a user's vote is what actually happened.
AgentX keeps three separate streams:
| Stream | Source | Where it lands |
| -------------------- | -------------------------------------------------------- | --------------------------------------------- |
| Operational outcomes | The trace itself (errors, failed tools, empty responses) | KPI failure metrics, Top failing |
| Scorer detections | Scorers you enabled | Signals, KPI metrics |
| **Human feedback** | **Your users' votes** | **Signals, Downvote rate, Judge Calibration** |
## Reporting a vote
From your app's vote handler, with the `trace_id` you kept from the traced agent call:
```python theme={null}
client.feedback.report(
trace_id=trace_id,
rating="down", # "up" or "down"
comment="It never answered my question", # optional - the user's own words
end_user_id=current_user.id, # optional - your identifier, opaque to AgentX
)
```
One call does four things:
1. **Attaches the vote to the trace** - an up/down vote chip in the trace dialog's header in
Observe, with the user's comment in the chip's tooltip.
2. **Raises a signal on a downvote** - a *Negative user feedback* signal lands in Monitor for
triage, tagged with a **User feedback** chip. The user is the detector: no sampling, no judge
call, nothing to configure or enable.
3. **Feeds Judge Calibration** - every vote (up and down) is recorded as an outcome report, so
AgentX's automated verdicts get measured against real human reactions. See
[Outcomes & Judge Calibration](/monitor/outcomes).
4. **Moves the Downvote rate KPI** - Overview's card shows the share of votes in the window
that were "down" (vote-denominated: of the users who reacted, how many were unhappy).
Reading votes back:
```python theme={null}
votes = client.feedback.list(trace_id) # the same rows the trace dialog shows as chips
kpis = client.monitor.kpis(window="7d") # includes downvoteRate and its delta vs the prior 7d
```
The wire equivalents are `POST /feedback` (`{"traceId", "rating", "comment?", "endUserId?"}`)
and `GET /feedback/trace/:traceId`.
## Feedback vs. outcomes
Both are ground truth; they differ in who reports. **Feedback** is a human vote with up/down
semantics, forwarded live from your UI. An **outcome** is an after-the-fact system result
("ticket reopened", reported by a workflow or webhook) with a free-form label. Both feed the
same calibration math, and neither ever counts as "AgentX flagged it in advance" - they are the
reports the judges get measured against, never predictions that inflate agreement.
## Try it
`sample-scripts/selfhost_demo/14_user_feedback.py` traces three simulated agent replies, votes
on them (two up, one down with a comment), and prints the resulting signal, the vote rows on the
trace, and the moved downvote rate - no LLM key needed.
# Quickstart
Source: https://developers.agentx.so/quickstart
Run the engine, open the dashboard, and see your first trace in five minutes
By the end of this page you'll have the engine running locally, the dashboard open, and a
real trace from your own code on screen.
```bash curl | bash theme={null}
curl -sSL https://raw.githubusercontent.com/AgentX-ai/AgentX-Trace-Eval/main/install.sh | bash
agentx-server --dev
```
```bash Python SDK theme={null}
pip install agentx-python
agentx-trace-eval --dev
```
```bash Docker theme={null}
git clone https://github.com/AgentX-ai/AgentX-Trace-Eval.git && cd AgentX-Trace-Eval
docker build -t agentx-selfhost .
docker run -d -p 4700:4700 -v agentx-data:/data agentx-selfhost
docker logs $(docker ps -lq) 2>&1 | grep "API key" # the key your SDK will use
```
The curl and pip paths download prebuilt binaries, and the Docker build compiles everything
inside the image - no local Node, Go, or Bun needed on any path. The startup log prints the
key you'll need in step 3:
```
AgentX self-host engine listening on http://localhost:4700
Default project API key: agtx_local_...
```
`--dev` opens `http://localhost:4700` in your browser (with Docker, open it yourself). In
the default no-login mode the engine hands the dashboard the Default project's API key
automatically, so you land directly on the **Overview** tab - nothing to paste. To confirm
you're connected, the corner of the **Settings** page shows the engine and dashboard
versions.
No login is the deliberate local-testing default: anyone who can reach the port gets the
key. For a shared or network-exposed engine, set `AGENTX_AUTH=enabled` to require sign-in -
see [Self-Host Configuration](/self-host/configuration).
Point the SDK at the engine with the same key:
```bash theme={null}
export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
export AGENTX_API_KEY=agtx_local_... # from the startup log
```
Then wrap your agent - a decorator for the simple case, a context manager for full control:
```python Decorator theme={null}
from agentx import AgentX
client = AgentX.from_env()
@client.tracer.trace("my-agent")
def answer(query: str) -> str:
# your agent logic - args become the input, the return value the output
return run_my_agent(query)
answer("How do I reset my password?")
client.tracer.flush(timeout=10) # send queued traces before a short script exits
```
```python Context manager theme={null}
from agentx import AgentX
client = AgentX.from_env()
query = "How do I reset my password?"
with client.tracer.trace("my-agent", input={"query": query}, model="gpt-4o-mini") as span:
answer = run_my_agent(query)
span.output = answer
client.tracer.flush(timeout=10)
```
Both snippets exit silently on success - tracing is fire-and-forget by design and never
raises into your agent. `flush()` returns `True` once the queued trace has been sent.
Using LangChain, OpenAI Agents, CrewAI, or another framework? A one-line integration captures
the full execution tree automatically - see [Framework Integrations](/sdk/integrations/langchain).
Open **Observe > Live Traces**. Your trace appears within seconds - click it for the full
detail: input/output, latency, tokens, estimated cost, and (for multi-step agents) the
Execution Timeline and Graph views of every step.
![PLACEHOLDER: screenshot of the trace detail dialog showing the stat strip (duration, tokens, cost), the Trajectory metrics row, and the Execution Timeline with an agent span, tool span, and LLM span nested under the root.]()
## Where to go next
Tool calls, sessions, span trees, async agents
Score your agent with an LLM judge in one script
Scorers and signals on live traffic
Block merges when quality drops
Provider keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY`)
unlock the LLM-judge features - evaluation scoring, live judge scorers,
semantic patterns. Set them as environment variables on the engine or later
from the dashboard's Settings page (a key set there takes precedence over
the env var). Tracing itself needs none.
# CI/CD Evaluation
Source: https://developers.agentx.so/sdk/ci-cd
Gate agent releases with eval test sets
The CI/CD evaluation feature lets you run your agent against a test dataset in your CI pipeline and receive a binary **PASS / FAIL** gate result. If the gate fails, the pipeline exits with a non-zero code and blocks the merge or deploy.
Two backends, two flows. `run_eval()` and the `create_ci_run` family below use the **hosted
platform's** CI Ingest API (`/ingest/ci-runs`) and 404 against a self-host engine. On
self-host, a CI run is an ordinary evaluation run plus the
[run gate](#gating-a-custom-agent-evaluation-run) - `report.gate(...)` /
`GET /runs/:id/gate` - documented in the
[API reference](/api-reference/ci-cd/overview).
Set `AGENTX_EVAL_QUIET=1` in CI to silence the interactive progress UI (spinners, per-case
lines) - gate verdicts, results, and errors still print, keeping a gated run's log to a few
lines.
## Prerequisites
1. Create an evaluation dataset in AgentX with at least one question.
2. Enable **CI/CD** in the dataset settings and set a pass rate threshold.
3. Export `AGENTX_API_KEY` in your environment.
## High-level: `run_eval()`
The easiest path: one call handles the entire lifecycle:
```python theme={null}
from agentx import AgentX, CIGateFailure
client = AgentX.from_env()
def my_agent(query: str) -> str:
# your agent here
return call_my_agent(query)
result = client.tracer.run_eval(
dataset_id="6876ddd222bbb333ccc444ee",
agent_fn=my_agent,
agent_name="customer-support-agent",
# Keys must be camelCase. The API's gitContext schema is strict
# and silently drops unrecognized (e.g. snake_case) keys.
git_context={
"branch": "feat/new-retrieval",
"commitSha": "a1b2c3d4e5f6",
},
fail_on_gate=True, # raises CIGateFailure if gate is "fail"
)
print(f"Gate: {result.gate} | Pass rate: {result.pass_rate:.0%}")
```
### Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ---------------------- | --------------- | ----------------------------------------------------------- |
| `dataset_id` | `str` | required | Dataset id; its grading config must have `ci.enabled: true` |
| `agent_fn` | `Callable[[str], str]` | required | Function that takes a query and returns a response |
| `agent_name` | `str` | none | Label for this agent on the AgentX platform |
| `pass_rate_threshold` | `float` | dataset default | Per-run override (0.0-1.0) |
| `git_context` | `dict` | none | Branch, commit SHA, PR number, etc. |
| `concurrency` | `int` | `1` | Max parallel question invocations |
| `fail_on_gate` | `bool` | `False` | Raise `CIGateFailure` if gate is `"fail"` |
| `timeout_per_question` | `float` | none | Seconds before a question times out |
### Return: `CIRunResult`
```python theme={null}
@dataclass
class CIRunResult:
run_id: str
gate: Literal["pass", "fail"]
pass_rate: float # e.g. 0.875
total_questions: int
passed_questions: int
scores: list[CIQuestionScore]
violations: list[ThresholdViolation]
finalized_at: str | None
```
## Low-level: step-by-step
For custom orchestration: parallel execution, streaming, or external agents:
```python theme={null}
from agentx import AgentX
from agentx.tracing import CIRun, CIRunResult
client = AgentX.from_env()
tracer = client.tracer
# 1. Create the run, receive test cases
run: CIRun = tracer.create_ci_run(
dataset_id="6876ddd222bbb333ccc444ee",
agent_name="my-agent",
git_context={"branch": "main", "commitSha": "abc1234"},
)
# 2. Run agent against each test case
for tc in run.test_cases:
output = my_agent(tc.query or "") # tc.query is None if exposeTestInputs is off
score = tracer.submit_result(
run.run_id,
tc.index,
output,
latency_ms=350,
)
if score.gate_fired:
print("failFast triggered, run already finalized as FAIL")
break
# 3. Finalize and get the gate decision
result: CIRunResult = tracer.finalize_ci_run(run.run_id)
print(f"Gate: {result.gate} ({result.passed_questions}/{result.total_questions} passed)")
```
## Exception handling
```python theme={null}
from agentx import AgentX, CIGateFailure, CINotEnabled, DatasetNotFound
try:
result = client.tracer.run_eval(
dataset_id=dataset_id,
agent_fn=my_agent,
fail_on_gate=True,
)
except DatasetNotFound:
print("Dataset not found, check the dataset_id")
sys.exit(1)
except CINotEnabled:
print("CI not enabled on this dataset, enable it in dataset settings")
sys.exit(1)
except CIGateFailure as e:
result = e.result
print(f"Gate FAILED, {result.pass_rate:.0%} passed")
for v in result.violations:
print(f" Q{v.question_index}: {v.metric} was {v.actual:.2f}, threshold {v.threshold:.2f}")
sys.exit(1)
```
## Parallel question execution
Run multiple questions concurrently to speed up large datasets:
```python theme={null}
result = client.tracer.run_eval(
dataset_id=dataset_id,
agent_fn=my_agent,
concurrency=4, # run 4 questions in parallel
fail_on_gate=True,
)
```
## Inspecting scores
```python theme={null}
for score in result.scores:
status = "✓" if score.passed else "✗"
print(f" [{status}] Q{score.question_index}: {score.rating}/10, {score.justification[:80]}")
```
## Polling a run
If you submit results asynchronously, poll until finalized:
```python theme={null}
import time
from agentx.tracing import CIRunStatus
while True:
status: CIRunStatus = tracer.get_ci_run(run_id)
if status.status in ("completed", "failed"):
break
time.sleep(5)
print(f"Gate: {status.gate}")
```
## GitHub Actions
See the [GitHub Actions integration guide](/integrations/github-actions) for a complete workflow template.
## Gating a Custom Agent Evaluation run
The flow above is the dedicated CI/CD run type (pass-rate gate over per-question pass/fail).
A regular [Custom Agent Evaluation](/sdk/evaluations/overview) run can be gated too, on its
average judge rating - useful when the same golden dataset serves both deep evaluation and the
merge gate (self-host):
```python theme={null}
import sys
run = (
client.evaluations
.run(dataset_id=DATASET_ID, subject=SUBJECT)
.execute(my_agent)
.finalize()
)
gate = run.gate(fail_under=7, no_regression=True)
sys.exit(gate.exit_code) # 0 on pass, 1 on fail
```
| Parameter | Type | Default | Description |
| --------------- | ------- | ------- | ---------------------------------------------------------------------------------------------- |
| `fail_under` | `float` | none | Fail when the run's average rating is below this floor (0-10) |
| `no_regression` | `bool` | `False` | Fail when the average dropped more than `tolerance` below the dataset's previous completed run |
| `tolerance` | `float` | `0.5` | Allowed drop for `no_regression` before it counts as a regression (judge scores are noisy) |
| `caller` | `str` | `"sdk"` | Free label recorded in the gate history, e.g. your CI job name |
At least one check is required. `gate()` prints a CI-log-friendly per-check verdict and
returns a `GateResult` with `.passed`, `.exit_code` (`0`/`1`), `.average_rating`,
`.baseline_average`, `.baseline_run_id`, and the per-check `.checks` list; the caller decides
whether to exit.
Every recorded verdict lands in the dashboard's CI Gates history. Two standalone forms cover
runs created elsewhere:
```python theme={null}
gate = client.evaluations.gate_run(run_id, fail_under=7, record=False) # preview, not recorded
client.evaluations.list_gates() # recorded verdicts, newest first
```
## pytest-native assertions
`agentx.testing.assert_evaluation` is the same gate as above in pytest ergonomics - a plain
`AssertionError` subclass, so it works in any runner with no plugin registration, and the
verdict still lands in the dashboard's CI gate history (`caller="pytest"`):
```python theme={null}
from agentx import AgentX
from agentx.testing import assert_evaluation
def test_support_agent_quality():
client = AgentX.from_env()
report = (
client.evaluations
.run(dataset_id=DATASET_ID, scorer_id=SCORER_ID, subject=SUBJECT)
.execute(my_agent)
.finalize()
)
assert_evaluation(report, min_rating=7.0, no_regression=True)
```
On failure the test output carries the per-check verdict ("\[FAIL] floor: average 5.1 below
fail\_under 7") plus the run id, so the red CI job links straight to the run in the dashboard.
# AI Analysis Report
Source: https://developers.agentx.so/sdk/evaluations/analysis-report
What .analyze() returns, strengths, weaknesses, instruction adherence, and recommendations
`.analyze()` is the last step in the evaluation chain. It runs the same durable, multi-stage pipeline as the dashboard's "Analyze" button: each response is scored by 1-3 LLM judges, then reduced through question- and cluster-level summaries into one final qualitative report, returned as a `Report` object.
```python theme={null}
report = client.evaluations.run(...).execute(my_agent).finalize().analyze(
mode="auto", # "auto" (default) | "sync" | "batch"
quality_mode="quality_first", # "quality_first" (default) | "balanced"
judges=["gpt-5.6-luna", "claude-opus-4-8"], # 1-3 model ids; omit for the platform default judge
)
report.summary # str | None, overall narrative summary
report.consistency_score # float | None, 0-10, run-to-run consistency
report.instruction_adherence # ReportInstructionAdherence | None
report.response_patterns # ReportResponsePatterns | None
report.reasoning_analysis # ReportReasoningAnalysis | None
report.tool_usage_analysis # ReportToolUsageAnalysis | None
report.strengths # list[str]
report.weaknesses # list[str]
report.overall_rating # str | None, "high" | "medium" | "low"
report.recommendations # list[ReportRecommendation]
```
Numeric scores (`average_rating`, similarity metrics) are ready right after `.finalize()`, so you don't need to wait for `.analyze()` just to see how the run went. Because `.analyze()` polls a job until it completes, it can take noticeably longer than a single LLM call for larger runs; progress is shown in the terminal while it waits.
## Controlling the analysis
| Parameter | Values | Description |
| --------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode` | `"auto"` (default), `"sync"`, `"batch"` | How item scoring executes server-side; `"auto"` picks based on run size |
| `quality_mode` | `"quality_first"`, `"balanced"` | `"quality_first"` runs a second judge on every item; `"balanced"` samples based on risk |
| `judges` | 1-3 model ids | Which LLM(s) score each response. The first always runs; a second confirms, a third only breaks a tie between the first two. Omit to score with a single judge, the engine's platform default model |
| `poll_interval` | seconds, default `5.0` | How often to check job status while waiting |
| `timeout` | seconds, default `1800.0` | Give up waiting after this long (the job keeps running server-side; call `get_report()` later to check on it) |
Check on a long-running analysis without calling `.analyze()` again, even from a separate script execution:
```python theme={null}
status = client.evaluations.get_analysis_status(run_id)
print(status.status, status.progress.overall_percentage)
```
## Fields
| Field | Shape | Description |
| ----------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `summary` | `str` | Overall narrative summary of the run |
| `consistency_score` | `float` | 0-10, how consistent responses were across repeated runs of the same question |
| `instruction_adherence` | `{ score, analysis, deviations: [str], rating }` | How well responses followed [`subject.agentInstructions`](/sdk/evaluations/evaluation-subject) |
| `response_patterns` | `{ similarities: [str], differences: [str], outliers: [str], rating }` | Cross-run consistency patterns |
| `reasoning_analysis` | `{ cot_quality, reasoning_patterns: [str], reasoning_gaps: [str], rating }` | Quality of the agent's chain-of-thought, when traced |
| `tool_usage_analysis` | `{ effectiveness, patterns: [str], issues: [str], rating }` | How well the agent used its tools, when tool calls were traced |
| `strengths` | `list[str]` | What the agent did well |
| `weaknesses` | `list[str]` | Where the agent fell short |
| `overall_rating` | `"high" \| "medium" \| "low"` | Coarse overall quality rating |
| `recommendations` | `[{ category, priority, recommendation, reasoning }]` | Actionable, prioritized fixes |
Each `rating` sub-field is one of `"high"`, `"medium"`, `"low"`. `recommendations[].category` is one of `"instructions"`, `"tools"`, `"knowledge"`, `"reasoning"`, `"consistency"`, `"other"`; `priority` is `"high"`, `"medium"`, or `"low"`.
## Reading it in your terminal
```python theme={null}
report = client.evaluations.run(...).execute(my_agent).finalize().analyze()
print(f"Rating: {report.overall_rating}")
print(report.summary)
for rec in report.recommendations:
print(f"[{rec.priority}] {rec.category}: {rec.recommendation}")
```
Or open `report.dashboard_url` to review the full report, per-question breakdowns, and instruction-change suggestions in the dashboard.
## Per-result rows without an analysis
`results()` returns the scored rows - no analysis job, no extra judge calls - for asserting
on individual results in scripts and CI. Rows are typed `RunResultRow` objects with snake\_case
attributes (dict-style access still works but is deprecated and warns; `row.raw` keeps the
full wire dict):
```python theme={null}
run = client.evaluations.run(...).execute(my_agent).finalize()
for r in run.results():
print(r.rating, r.question_text, r.justification)
for scorer in r.code_scorer_results or []: # code scorers, trajectory/context match
print(" ", scorer["name"], scorer["score"])
```
Each row carries `rating`/`justification`, a `status` (`"scored"`, `"skipped"` when the judge
could not score it - rating stays `None`, not 0 - or `"failed"` when the result carried an
error), the linked `trace_id`, `latency_ms` and token counts, and the similarity metrics
(`cosine_similarity`, `jaccard_similarity`, `bleu_score`, `rouge_score`).
`client.evaluations.get_run(run_id)` fetches the same payload later, from a different process.
# Build Dataset
Source: https://developers.agentx.so/sdk/evaluations/build-dataset
Create evaluation datasets and test cases with the Python SDK
A dataset is a named set of test cases plus scoring configuration. Every evaluation run locks a snapshot of the dataset at creation time, so editing a dataset later never changes the scoring of past runs.
There are two ways to build one: the fluent builder, or importing from a CSV.
## Programmatic builder
```python theme={null}
dataset = (
client.evaluations.datasets
.builder(
name="Support Agent v2",
number_of_requests=3,
acceptance_criteria="Accurate, concise, grounded in docs.",
rejection_criteria="No hallucinated policies.",
)
.add_case(
query="How do I reset my password?",
expected_results="Explain the password reset process step by step.",
)
.add_case(
query="What payment methods do you accept?",
expected_results="List supported payment methods clearly.",
)
.publish()
)
print(dataset.id)
```
### `builder()` parameters
| Parameter | Type | Default | Description |
| --------------------- | ------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | required | Dataset display name |
| `description` | `str` | none | Human-readable description |
| `number_of_requests` | `int` | `1` | How many times each question is run per evaluation, for consistency testing |
| `acceptance_criteria` | `str` | none | What a good response looks like, included in the LLM scoring prompt |
| `rejection_criteria` | `str` | none | What a bad response looks like, included in the LLM scoring prompt |
| `evaluation_criteria` | `str` | none | Scoring rubric, included in the LLM scoring prompt |
| `vector_similarity` | `bool` | `False` | Enable cosine similarity scoring against `expected_results` |
| `jaccard_similarity` | `bool` | `False` | Enable Jaccard token-overlap scoring against `expected_results` |
| `bleu_score` | `bool` | `False` | Enable BLEU scoring against `expected_results` |
| `rouge_score` | `bool` | `False` | Enable ROUGE-L scoring against `expected_results` |
| `similarity_model` | `str` | provider default | Embedding model used for vector similarity, e.g. `"text-embedding-3-small"` |
| `sovereignty_models` | `list[str]` | none | Enables Sovereignty & Portability: models to compare on this dataset. Discover valid ids with `client.evaluations.list_models()` |
| `code_scorers` | `list[dict]` | none | Deterministic JS scorers run per result alongside the judge; each entry is `{"name", "code"}` plus optional `"enabled"` (default `True`). See [Code scorers](/evaluation/code-scorers) |
| `judge_prompt` | `str` | none | Override the LLM-as-judge's raw prompt template. See [Configuring the judge](/sdk/evaluations/evaluation-settings#configuring-the-judge) |
| `judge_model` | `str` | none | Override the LLM-as-judge's model (any OpenAI or Anthropic id from `client.evaluations.list_models()`) |
### Repetition statistics
With `number_of_requests` above 1, the run reports `caseStatistics`: per case, the
`averageRating`, `minRating`, `maxRating`, and `ratingVariance` across the repetitions (cases
need at least two rated repetitions; smoke-test variants are excluded). A case swinging between
3 and 9 is a different problem from a case stuck at 6, and only the variance tells them apart.
Connector-driven runs - the dashboard invoking your connected agent endpoint - honor the
dataset's `numberOfRequests` too, clamped to 1-10.
### `add_case()` parameters
| Parameter | Type | Description |
| ---------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `query` | `str` | required, the question or prompt sent to the agent |
| `expected_results` | `str` | Optional, enables vector and Jaccard similarity scoring for this case |
| `expected_capabilities` | `list[str]` | Optional. Capabilities/tools a correct response should demonstrate, shown to the judge |
| `expected_knowledge_base` | `list[str]` | Optional. Knowledge sources a correct response should draw on, shown to the judge |
| `expected_delegations` | `list[str]` | Optional. Sub-agents a correct run should delegate to, shown to the judge |
| `follow_up_questions` | `list[dict]` | Optional. Follow-up turns for this case, each a dict with the same keys as a main question |
| `judge_guideline` | `str` | Optional extra grading instructions for this specific case, layered on top of the dataset's criteria fields |
| `smoke_test_count` | `int` | Optional, 1-10. Ask this question that many extra ways each run (LLM-paraphrased server-side), to catch agents that break on phrasing rather than substance. See [Smoke testing](#smoke-testing) below |
| `smoke_test_guidance` | `str` | Optional, only used when `smoke_test_count` is set. Free text steering what kind of variants get generated (tone, adversarial phrasing, different languages, ...) |
| `expected_tools` | `list[str]` | Optional. The tool calls a correct run of this case should make - trajectory-matched against the result's linked trace, reported as a pass/fail "Trajectory match" scorer row. See [Expected trajectories](#expected-trajectories) below |
| `trajectory_match_mode` | `str` | `"strict"` (default), `"unordered"`, `"superset"`, or `"subset"` - see below |
| `expected_retrieval_context` | `str \| list[str]` | Optional. The chunk(s) a correct retriever should fetch for this query - compared against the actually retrieved context with token-level Jaccard similarity, reported as a "Context match (jaccard)" scorer row (0-1), no judge call. See [Expected retrieval context](#expected-retrieval-context) below |
| `splits` | `list[str]` | Optional. Named subsets this case belongs to, e.g. `["smoke"]` - a run can then target just one split. See [Splits](#splits) below |
`publish()` sends the dataset to AgentX and returns a `Dataset` object with `.id`, which you pass as `dataset_id` to `client.evaluations.run()`.
## Expected trajectories
Agentic cases can pin down *how* the answer should be produced, not just what it should say.
Declare the expected tool calls; when a result links its trace (return
`{"output": ..., "trace_id": span.trace_id}` from your agent function), the engine compares the
trace's actual tool-call sequence deterministically:
```python theme={null}
.add_case(
query="I want a refund for order A-1001, it arrived two weeks ago.",
expected_results="Confirms eligibility within the 30-day window.",
expected_tools=["lookup_order", "refund_policy"],
trajectory_match_mode="unordered",
)
```
| Mode | Passes when |
| ----------- | ------------------------------------------------ |
| `strict` | Same calls, same order |
| `unordered` | Same calls, any order |
| `superset` | Every expected call present; extra calls allowed |
| `subset` | No unexpected calls; missing some is allowed |
The verdict lands on each result as a named scorer row alongside the judge rating - and the LLM
judge itself also sees the full trajectory in its prompt, so written criteria about tool
behavior are scoreable too.
## Expected retrieval context
RAG cases can pin what a correct retriever *should have fetched*. At scoring time the engine
compares it against the actually retrieved context - the `retrieval_context` your agent
function returned, else the linked trace's retrieval spans - using token-level Jaccard
similarity, and reports a **Context match (jaccard)** scorer row (0-1) on each result. It's
deterministic and costs no judge call, so it's safe to run on every retriever, chunking, or
embedding change:
```python theme={null}
.add_case(
query="What is the refund window?",
expected_retrieval_context=[
"Refund policy: refunds are available within 30 days of delivery for unused items."
],
)
```
The same field is editable in the dashboard (open a dataset case, **Expected retrieval
context**, blank line between chunks). Jaccard measures token overlap, not meaning - pair it
with the [RAG judge metrics](/evaluation/rag) for semantic verdicts.
## Smoke testing
Every variant of a smoke-tested case is graded against the same `expected_results` as the original question. Both the paraphrase text and the count are generated entirely server-side, `.execute()` asks the extra variants and submits them automatically:
```python theme={null}
dataset = (
client.evaluations.datasets
.builder(name="Support Agent v2")
.add_case(
query="How do I reset my password?",
expected_results="Explain the password reset process step by step.",
smoke_test_count=3,
smoke_test_guidance="try terse, frustrated, and non-native-speaker phrasing",
)
.publish()
)
```
Your agent function can check `case.is_smoke_test_variant` if it wants to handle variants differently, but this isn't required, `case.query` already holds the text to ask either way. Smoke-test results are scored normally but reported as a separate robustness signal, excluded from `report.average_rating` so one hard phrasing doesn't skew your headline score.
Only a case's opening question can be smoke-tested, `smoke_test_count`/`smoke_test_guidance` are ignored on follow-up questions.
## Splits
One golden dataset usually serves two budgets: a handful of cases on every PR, everything on a
schedule. Tag cases into named subsets and pick the subset at run time instead of maintaining
two datasets:
```python theme={null}
dataset = (
client.evaluations.datasets
.builder(name="Support Agent v2")
.add_case(query="How do I reset my password?", expected_results="...", splits=["smoke"])
.add_case(query="What payment methods do you accept?", expected_results="...")
.publish()
)
client.evaluations.run(dataset.id, subject, split="smoke").execute(my_agent).finalize()
```
A run with `split="smoke"` executes only the tagged cases; omit `split` to run everything. The
dashboard's Run dialog and connector-driven runs take the same split. Case indexes are
preserved across split and full runs, so per-case history lines up whichever subset ran.
## CSV import
`from_csv()` returns the same builder as above with one case per row, so you can keep chaining
(`.add_case(...)`) and must finish with `.publish()`:
```python theme={null}
dataset = client.evaluations.datasets.from_csv(
"support_questions.csv",
name="Support Agent v2",
acceptance_criteria="Accurate, concise, grounded in docs.",
).publish()
```
The only required column is `query`. Optional columns: `expected_results`, plus
`expected_capabilities`, `expected_knowledge_base`, and `expected_delegations`
(semicolon-separated lists). Rows with an empty `query` are skipped with a logged warning.
Any extra keyword arguments (criteria, similarity metrics, `judge_model`, ...) are passed
through to `builder()`.
```csv theme={null}
query,expected_results
How do I reset my password?,Explain the password reset process step by step.
What payment methods do you accept?,List supported payment methods clearly.
```
`client.evaluations.datasets.from_dataframe(df, name=...)` does the same from a pandas
DataFrame with the same column contract.
## Next steps
Run an evaluation against your new dataset
Reuse one grading config across multiple datasets
Describe the agent under test
# Evaluation Settings
Source: https://developers.agentx.so/sdk/evaluations/evaluation-settings
Build a standalone, reusable grading config decoupled from any one dataset
A dataset's questions and its grading config (acceptance criteria, similarity metrics, thresholds) are independent. `EvaluationSettings` is the grading config half on its own: build it once, then run it against any dataset by id, instead of every dataset carrying its own copy.
Since the judge-scorer unification, an evaluation-settings record IS the judge rubric + offline profile of an [LLM Judge Scorer](/monitor/online-evaluators) - the unified entity at `client.monitor.judge_scorers`, which also carries the optional online (live-traffic) profile. This surface keeps working unchanged (it now emits a default-hidden `DeprecationWarning` on first use); prefer `judge_scorers` for new code so both profiles live in one place - `client.monitor.judge_scorers.builder(...)` takes the same snake\_case parameters as the builder below, plus `tool_context`, `thresholds`, and the live profile in the same call - and manage everything from the dashboard's Scorers page.
Passing a grader id is entirely optional (`scorer_id` is the current name; `evaluation_settings_id` is the pre-consolidation alias for the same id). Omit it and a run grades against the dataset's own bundled config, exactly as if this feature didn't exist. On self-host, `additional_scorer_ids=[...]` adds up to 4 extra judges that each score every result too - the primary keeps the `rating` column, extra verdicts land in `judge_scorer_results` per row (see the [quickstart](/sdk/evaluations/quickstart#multiple-judge-scorers-per-run)).
## Programmatic builder
```python theme={null}
evaluation_settings = client.evaluations.settings.builder(
name="Strict - production bar",
number_of_requests=3,
acceptance_criteria="Accurate, concise, grounded in the support policy.",
rejection_criteria="No hallucinated policies or made-up steps.",
vector_similarity=True,
jaccard_similarity=True,
).publish()
print(evaluation_settings.id)
```
### `builder()` parameters
| Parameter | Type | Default | Description |
| --------------------- | ------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name` | `str` | required | Config display name |
| `description` | `str` | none | Human-readable description |
| `number_of_requests` | `int` | `1` | How many times each question is run per evaluation, for consistency testing |
| `acceptance_criteria` | `str` | none | What a good response looks like, included in the LLM scoring prompt |
| `rejection_criteria` | `str` | none | What a bad response looks like, included in the LLM scoring prompt |
| `evaluation_criteria` | `str` | none | Scoring rubric, included in the LLM scoring prompt |
| `vector_similarity` | `bool` | `False` | Enable cosine similarity scoring against `expected_results` |
| `jaccard_similarity` | `bool` | `False` | Enable Jaccard token-overlap scoring against `expected_results` |
| `bleu_score` | `bool` | `False` | Enable BLEU scoring against `expected_results` |
| `rouge_score` | `bool` | `False` | Enable ROUGE scoring against `expected_results` |
| `similarity_model` | `str` | provider default | Embedding model used for vector similarity, e.g. `"text-embedding-3-small"` |
| `sovereignty_models` | `list[str]` | none | Enables Sovereignty & Portability; models to compare when this config runs. Discover valid ids with `client.evaluations.list_models()` |
| `code_scorers` | `list[dict]` | none | Deterministic JS scorers run per result alongside the judge; each entry is `{"name", "code", "enabled"}`. See [Code scorers](/evaluation/code-scorers) |
| `judge_prompt` | `str` | server default | Override the LLM-as-judge's raw prompt template. See [Configuring the judge](#configuring-the-judge) |
| `judge_model` | `str` | `"gpt-5.6-luna"` | Override the LLM-as-judge's model (any OpenAI or Anthropic id from `client.evaluations.list_models()`) |
`publish()` sends the config to AgentX and returns an `EvaluationSettings` object with `.id`, which you pass as `scorer_id` (or the alias `evaluation_settings_id`) to `client.evaluations.run()`.
With `number_of_requests` above 1, runs report per-case repetition spread as `caseStatistics` -
see [Repetition statistics](/sdk/evaluations/build-dataset#repetition-statistics).
## Configuring the judge
`judge_prompt`/`judge_model` apply to every scoring path, native dashboard runs and SDK/custom-agent runs alike:
```python theme={null}
evaluation_settings = client.evaluations.settings.builder(
name="Strict - production bar",
judge_model="claude-opus-4-8",
judge_prompt="""You are grading a customer support response.
**User Query:** {input}
**Agent Response:**
{output}
**Expected Results:**
{expected}
Score strictly: any missing policy detail is a failing response.""",
).publish()
```
`judge_prompt` is a raw template: `{input}`, `{output}`, and `{expected}` are substituted in. Everything else the judge needs (chain of thought, capabilities/references, criteria, per-question `judge_guideline`, delegation notes) is appended automatically after it, so a custom prompt can restructure the grading philosophy without ever losing that context. Omit `judge_prompt` to keep the default rubric, or `judge_model` to keep the default (`gpt-5.6-luna`).
## Running against a dataset
```python theme={null}
report = (
client.evaluations
.run(
dataset_id="...",
subject={"kind": "custom_agent", "displayName": "Support Bot", "framework": "langchain", "runtime": "local"},
evaluation_settings_id=evaluation_settings.id,
)
.execute(my_agent)
.finalize()
.analyze()
)
```
The same `evaluation_settings_id` can be reused across as many `dataset_id`s as you want. Build it once, grade any dataset with it going forward.
## Fetching an existing config
```python theme={null}
evaluation_settings = client.evaluations.settings.get("6876eee333ccc444ddd555ff")
all_configs = client.evaluations.settings.list()
```
## Next steps
Create the test cases this config will grade
Run an evaluation end to end
# EvaluationSubject Fields
Source: https://developers.agentx.so/sdk/evaluations/evaluation-subject
Describe the agent being evaluated so analysis can check instruction adherence
`subject` (passed to `client.evaluations.run()`) tells AgentX what it's evaluating. It's used by AI analysis to check instruction adherence and to power the Sovereignty & Portability matrix; it never changes how scoring itself runs.
```python theme={null}
report = (
client.evaluations
.run(
dataset_id="...",
subject={
"kind": "custom_agent",
"displayName": "Support Bot",
"framework": "langchain",
"runtime": "local",
"agentInstructions": "You are a helpful support agent...",
},
)
.execute(my_agent)
.finalize()
.analyze()
)
```
## Fields
| Field | Values | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| `kind` | `"custom_agent"` | Always `"custom_agent"` for externally-owned agents evaluated via this SDK |
| `displayName` | `str` | Human-readable identifier shown in the dashboard |
| `framework` | `"raw_python"` \| `"openai"` \| `"anthropic"` \| `"google"` \| `"langchain"` \| `"llamaindex"` \| `"crewai"` \| `"autogen"` \| `"n8n"` \| `"flowise"` \| `"other"` | Underlying technology stack |
| `runtime` | `"local"` \| `"ci"` \| `"customer_hosted"` \| `"low_code"` | Where the agent is running. Defaults to `"local"` |
| `frameworkVersion` | `str` | Optional framework version identifier, e.g. `"0.3.14"` |
| `agentInstructions` | `str` | Optional system prompt / instructions, checked against actual behavior during analysis |
| `metadata` | `dict[str, str \| int \| bool]` | Optional free-form tags recorded on the run |
Every field is optional: `kind` defaults to `"custom_agent"` and `runtime` to `"local"`. Set at least `displayName` and `framework` so the dashboard can label the run; use `"other"` when your stack isn't in the `framework` list (any value outside it is rejected - see the note on the [OpenAI Agents SDK example](/sdk/evaluations/examples/openai-agents)). Snake\_case keys (`display_name`, `agent_instructions`, ...) are accepted too.
## Return values from your agent function
Whatever callable you pass to `.execute()` can return:
* **A string**: used directly as the output text.
* **A dict**: `"output"` (or `"text"`/`"response"`) is the output text. Recognized optional keys: `"trace_id"` links the result to a recorded trace (enabling View trace, trajectory-aware judging, and [expected-trajectory matching](/sdk/evaluations/build-dataset#expected-trajectories)); `"retrieval_context"` (string or chunk list) feeds [RAG judging](/evaluation/rag); `"metadata"` (dict) is stored on the result; `"input_tokens"`/`"output_tokens"` fill in the timing detail; `"trace"` attaches a lightweight `{"events": [...]}` observable trace; `"error"` marks the case as failed.
* **An `EvaluationResult`**: full manual control, including the observable trace and timings. Case identity fields are required by the model (the SDK re-stamps them from the case anyway):
```python theme={null}
from agentx.evaluations.models import EvaluationResult, ResultTimings
def my_agent(case):
return EvaluationResult(
case_id=case.case_id,
question_index=case.question_index,
run_number=case.run_number,
input={"query": case.query},
output={"text": "Click Forgot Password on the login screen."},
metadata={"model": "gpt-4o"},
timings=ResultTimings(latency_ms=1240, input_tokens=150, output_tokens=45),
)
```
For most agents the dict form is all you need; reach for `EvaluationResult` only when you want to hand-build the observable trace or error object.
The SDK uploads only what your callable returns, plus the case's `query` as the result's input. Nothing else from your process (environment, prompts you didn't return, other variables) is sent.
# Anthropic Claude
Source: https://developers.agentx.so/sdk/evaluations/examples/anthropic
Evaluate an agent built on the Anthropic Messages API
Install:
```bash theme={null}
pip install agentx-python anthropic
```
## Usage
```python theme={null}
from agentx import AgentX
import anthropic
client = AgentX.from_env()
ant = anthropic.Anthropic()
def claude_agent(case):
msg = ant.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
system="You are a helpful customer support agent.",
messages=[{"role": "user", "content": case.query}],
)
return {"output": msg.content[0].text, "metadata": {"model": msg.model}}
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Claude Haiku", "framework": "anthropic"})
.execute(claude_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
Returning `metadata: {"model": msg.model}` records which model produced each response, powering the Sovereignty & Portability breakdown in the report.
## With tracing
Combine this with [`patch_anthropic_client`](/sdk/integrations/anthropic) to get a full Execution Timeline per result, not just a score. Wrap the patched call in `tracer.trace(..., sync=True)` so `span.trace_id` is populated before your function returns. The patch attaches to that span instead of sending its own independent trace, so you still get exactly one trace per case:
```python theme={null}
from agentx import AgentX
from agentx.integrations.anthropic import patch_anthropic_client
import anthropic
client = AgentX.from_env()
ant = anthropic.Anthropic()
patch_anthropic_client(ant, tracer=client.tracer, name="claude-support-agent")
def claude_agent(case):
with client.tracer.trace("claude-support-agent-call", framework="anthropic", sync=True, monitor=False) as span:
msg = ant.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
system="You are a helpful customer support agent.",
messages=[{"role": "user", "content": case.query}],
)
span.output = msg.content[0].text
return {
"output": msg.content[0].text,
"metadata": {"model": msg.model},
"trace_id": span.trace_id,
}
```
`patch_anthropic_client` on its own (no surrounding span) is fire-and-forget and never returns a `trace_id`; see [Decorator vs. context manager](/sdk/tracing#decorator-vs-context-manager). Wrapping it in `tracer.trace(..., sync=True)`, as shown above, is what makes the id available.
A complete working example is available as [`anthropic_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/anthropic_eval.py) in the AgentX-Python repository.
# AutoGen / AG2
Source: https://developers.agentx.so/sdk/evaluations/examples/autogen
Evaluate a conversation between AutoGen (or AG2) ConversableAgents
Install:
```bash theme={null}
pip install agentx-python pyautogen
```
AG2 is the actively-maintained fork of AutoGen. If you're on AG2, install `ag2` instead of `pyautogen`; the `autogen` import name and API shown below are the same either way.
## Usage
```python theme={null}
from agentx import AgentX
from autogen import ConversableAgent
client = AgentX.from_env()
assistant = ConversableAgent(
name="assistant",
system_message="You are a helpful customer support agent.",
llm_config={"config_list": [{"model": "gpt-4o-mini"}]},
)
user_proxy = ConversableAgent(
name="user_proxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=1,
llm_config=False,
)
def autogen_agent(case):
chat_result = user_proxy.initiate_chat(
assistant,
message=case.query,
max_turns=1,
)
output = chat_result.chat_history[-1]["content"]
return {"output": output, "metadata": {"model": "gpt-4o-mini"}}
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Support Assistant", "framework": "autogen"})
.execute(autogen_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
Returning `metadata: {"model": ...}` records which model produced each response, powering the Sovereignty & Portability breakdown in the report. For a full Execution Timeline, use the [AutoGen tracing integration](/sdk/integrations/autogen) (steps named by the speaking agent, tool calls captured), or wrap calls in [`tracer.trace(..., sync=True)`](/sdk/tracing#linking-a-trace-to-an-evaluation-result) yourself, the same way the [OpenAI example](/sdk/evaluations/examples/openai) does.
A complete working example, including a multi-agent `GroupChat` variant with a researcher/writer split, is available as [`autogen_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/autogen_eval.py) in the AgentX-Python repository.
# CrewAI
Source: https://developers.agentx.so/sdk/evaluations/examples/crewai
Evaluate a CrewAI crew
Install:
```bash theme={null}
pip install agentx-python crewai
```
## Usage
```python theme={null}
from agentx import AgentX
from crewai import Agent, Task, Crew
client = AgentX.from_env()
support_agent = Agent(
role="Support Specialist",
goal="Resolve customer questions accurately and concisely",
backstory="You are an experienced customer support specialist.",
)
def crewai_agent(case):
task = Task(
description=case.query,
expected_output="A clear, accurate answer to the customer's question.",
agent=support_agent,
)
crew = Crew(agents=[support_agent], tasks=[task])
result = crew.kickoff()
return {"output": str(result)}
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Support Crew", "framework": "crewai"})
.execute(crewai_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
## With tracing
Use [`AgentXCrewObserver`](/sdk/integrations/crewai) to get a full Execution Timeline per result. Its `observer.kickoff(...)` convenience wrapper is fire-and-forget and can't return a `trace_id`. Use `observer.observe(sync=True)` instead, which blocks until the trace is ingested:
```python theme={null}
from agentx import AgentX
from agentx.integrations.crewai import AgentXCrewObserver
from crewai import Agent, Task, Crew
client = AgentX.from_env()
observer = AgentXCrewObserver(client.tracer, name="support-crew")
support_agent = Agent(
role="Support Specialist",
goal="Resolve customer questions accurately and concisely",
backstory="You are an experienced customer support specialist.",
)
def crewai_agent(case):
task = Task(
description=case.query,
expected_output="A clear, accurate answer to the customer's question.",
agent=support_agent,
)
crew = Crew(agents=[support_agent], tasks=[task])
with observer.observe(input={"query": case.query}, sync=True) as span:
result = crew.kickoff()
span.output = str(result)
return {"output": str(result), "trace_id": span.trace_id}
```
`observer.observe()` doesn't auto-populate per-task tool calls the way `kickoff()` does. Record them yourself with `span.add_tool_call(...)` if you need that detail. See the [CrewAI tracing integration](/sdk/integrations/crewai) for the full comparison. It also takes no `monitor` flag - traces created inside `.execute()` are stamped `monitor=False` automatically by the eval-run scope, so the run's own judge stays the only scorer.
A complete working example is available as [`crewai_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/crewai_eval.py) in the AgentX-Python repository.
# HTTP Endpoint
Source: https://developers.agentx.so/sdk/evaluations/examples/http-endpoint
Evaluate an agent exposed as a web service, without writing a Python callable
No install needed beyond `agentx-python` itself. `HttpEndpointAdapter` uses `requests`, already a core dependency. Use it when your agent runs behind an HTTP service (FastAPI, LangServe, Flask, an n8n webhook, or anything else) instead of writing a Python function.
## Usage
```python theme={null}
from agentx import AgentX
from agentx.evaluations.adapters.http_endpoint import HttpEndpointAdapter
client = AgentX.from_env()
adapter = HttpEndpointAdapter(
url="http://localhost:8000/agent/invoke",
headers={"Authorization": "Bearer your-token"},
timeout=30,
)
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "My API Agent", "framework": "other"})
.execute(adapter)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
### Endpoint contract
For every test case, the adapter sends:
```
{method} {url}
Content-Type: application/json
{
"query": "How do I reset my password?",
"case_id": "case-0",
"question_index": 0,
"run_number": 1
}
```
Your service must respond with a `2xx` and a JSON body. A non-2xx status or a request timeout is recorded as a failed result for that case (status `failed`, no rating), not a hard error that stops the run.
| Response field | Required | Description |
| ----------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output` (or `text`, or `response`) | Yes | The agent's response text |
| `metadata` | No | Arbitrary key-value data. Include `model` to power the Sovereignty & Portability breakdown |
| `trace` | No | `{ "events": [...] }`, a lightweight reasoning/tool-call summary shown in AI analysis. For a full browsable Execution Timeline instead, have your service submit a real trace itself (`POST /ingest/traces`, or the [tracing SDK](/sdk/tracing)) and return `"trace_id"` here |
| `retrieval_context` | No | String or chunk list: what the agent actually retrieved for this case, feeds [RAG judging](/evaluation/rag) and expected-retrieval-context matching |
| `input_tokens` / `output_tokens` | No | Token counts, shown in the result's timing detail |
```json theme={null}
{
"output": "Click Forgot Password on the login screen.",
"metadata": { "model": "gpt-4o-mini" },
"trace_id": "6876fff444ggg555hhh666jj"
}
```
### `HttpEndpointAdapter` parameters
```python theme={null}
HttpEndpointAdapter(
url: str,
headers: dict | None = None,
timeout: int = 30,
method: str = "POST",
)
```
| Parameter | Type | Default | Description |
| --------- | ------ | -------- | -------------------------------------------------------------------------------- |
| `url` | `str` | required | Endpoint to call for each test case |
| `headers` | `dict` | `None` | Extra headers, e.g. an `Authorization` bearer token |
| `timeout` | `int` | `30` | Seconds to wait for a response before recording the case as failed |
| `method` | `str` | `"POST"` | HTTP method to use, override if your service expects something other than `POST` |
The SDK (running locally) makes the request directly to your service. The AgentX API never touches your endpoint. Your endpoint must be reachable from wherever the script runs, not from AgentX's servers.
A complete working example is available as [`http_endpoint_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/http_endpoint_eval.py) in the AgentX-Python repository.
# LangChain
Source: https://developers.agentx.so/sdk/evaluations/examples/langchain
Evaluate a LangChain chain or agent
Install:
```bash theme={null}
pip install agentx-python langchain langchain-openai
```
## Usage
```python theme={null}
from agentx import AgentX
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
client = AgentX.from_env()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful support agent."),
("human", "{question}"),
])
chain = prompt | llm | StrOutputParser()
def langchain_agent(case):
output = chain.invoke({"question": case.query})
return {"output": output, "metadata": {"model": "gpt-4o-mini"}}
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Support Chain", "framework": "langchain"})
.execute(langchain_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
Returning `metadata: {"model": ...}` records which model produced each response, powering the Sovereignty & Portability breakdown in the report. This pattern works the same way for `AgentExecutor`-style agents: call `agent.invoke(...)` instead of `chain.invoke(...)` inside `langchain_agent`.
## With tracing
Pass an `AgentXCallbackHandler` into `chain.invoke(..., config={"callbacks": [handler]})` to get a full Execution Timeline per result. Unlike the raw-tracer pattern used for OpenAI/Anthropic, the handler's traces are always sent async, so `trace_id` isn't available directly from it. Capture it by additionally wrapping the call in a `sync=True` span, the same way the [Anthropic example](/sdk/evaluations/examples/anthropic#with-tracing) does:
```python theme={null}
from agentx import AgentX
from agentx.integrations.langchain import AgentXCallbackHandler
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
client = AgentX.from_env()
handler = AgentXCallbackHandler(tracer=client.tracer, name="support-chain")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful support agent."),
("human", "{question}"),
])
chain = prompt | llm | StrOutputParser()
def langchain_agent(case):
with client.tracer.trace("support-chain-call", framework="langchain", sync=True, monitor=False) as span:
output = chain.invoke({"question": case.query}, config={"callbacks": [handler]})
span.output = output
return {
"output": output,
"metadata": {"model": "gpt-4o-mini"},
"trace_id": span.trace_id,
}
```
If you only want tracing (not evaluation), see the [LangChain tracing integration](/sdk/integrations/langchain): pass `AgentXCallbackHandler` on its own with no surrounding span, and every chain invocation is traced automatically.
A complete working example is available as [`langchain_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/langchain_eval.py) in the AgentX-Python repository.
# LlamaIndex
Source: https://developers.agentx.so/sdk/evaluations/examples/llamaindex
Evaluate a LlamaIndex RAG query engine
Install:
```bash theme={null}
pip install agentx-python llama-index llama-index-llms-openai
```
## Usage
```python theme={null}
from agentx import AgentX
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
client = AgentX.from_env()
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(llm=OpenAI(model="gpt-4o-mini"))
def llamaindex_agent(case):
response = query_engine.query(case.query)
return {"output": str(response), "metadata": {"model": "gpt-4o-mini"}}
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Docs RAG Agent", "framework": "llamaindex"})
.execute(llamaindex_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
Returning `metadata: {"model": ...}` records which model produced each response, powering the Sovereignty & Portability breakdown in the report. For a full Execution Timeline, use the [LlamaIndex tracing integration](/sdk/integrations/llamaindex) (its instrumentation handler captures LLM calls, retrievals, and tool calls automatically), or wrap calls in [`tracer.trace(..., sync=True)`](/sdk/tracing#linking-a-trace-to-an-evaluation-result) yourself. `response.source_nodes` (for a query engine) or `response.sources` (for an agent) are worth recording as tool calls or retrieval steps on that span.
A complete working example, with a RAG-vs-ReAct-agent toggle and per-source retrieval tracing, is available as [`llamaindex_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/llamaindex_eval.py) in the AgentX-Python repository.
# OpenAI
Source: https://developers.agentx.so/sdk/evaluations/examples/openai
Evaluate a plain OpenAI chat completions agent
Install:
```bash theme={null}
pip install agentx-python openai
```
## Usage
```python theme={null}
from agentx import AgentX
from openai import OpenAI
client = AgentX.from_env()
oai = OpenAI()
def openai_agent(case):
resp = oai.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful support agent."},
{"role": "user", "content": case.query},
],
)
return {"output": resp.choices[0].message.content, "metadata": {"model": resp.model}}
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "GPT-4o-mini", "framework": "openai"})
.execute(openai_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
Returning `{"output": ..., "metadata": {"model": resp.model}}` records which model produced each response, powering the Sovereignty & Portability breakdown in the report.
## With tracing
Wrap the call in `tracer.trace(..., sync=True)` to link a trace to each result. (For always-on production tracing there is also [`patch_openai_client`](/sdk/integrations/openai), which auto-traces every `chat.completions.create()` call - but its traces are fire-and-forget and can't hand you a `trace_id`, so the explicit span below is the pattern for evaluations.) `sync=True` blocks until AgentX has ingested the trace, so `span.trace_id` is populated by the time the `with` block exits. The default (fire-and-forget) mode never learns the trace\_id. Returning `trace_id` alongside `output` is what makes the result's "Message Trace Details → Execution Timeline" viewable in the dashboard, not just the score:
```python theme={null}
from agentx import AgentX
from openai import OpenAI
client = AgentX.from_env()
oai = OpenAI()
def openai_agent(case):
with client.tracer.trace(
"openai-agent-call",
input={"query": case.query},
framework="openai",
model="gpt-4o-mini",
sync=True,
monitor=False, # the run's judge already scores this case
) as span:
resp = oai.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful support agent."},
{"role": "user", "content": case.query},
],
)
span.output = resp.choices[0].message.content
return {
"output": resp.choices[0].message.content,
"metadata": {"model": resp.model},
"trace_id": span.trace_id,
}
```
`trace_id` is optional; everything else about the run is unchanged whether or not you include it.
## Full example
Everything together: building a dataset, publishing a reusable grading config, running the agent with tracing, and reading results at every stage:
```python theme={null}
import os
from typing import Any, Dict
from openai import OpenAI
from agentx import AgentX
from agentx.evaluations.models import Dataset, EvaluationCase, Report
from agentx.evaluations.runner import EvaluationRunContext
from agentx.monitor.judge_scorers import JudgeScorer
client = AgentX(
api_key=os.environ["AGENTX_API_KEY"],
workspace_id=os.environ.get("AGENTX_WORKSPACE_ID"), # set if your key spans multiple workspaces
)
oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
# 1. Build a dataset. Skip this and pass an existing dataset_id if you already have one.
dataset: Dataset = (
client.evaluations.datasets.builder(
name="Support Agent Eval Sample",
number_of_requests=2, # runs per case
acceptance_criteria="Accurate, concise, grounded in the support policy.",
rejection_criteria="No hallucinated policies or made-up steps.",
)
.add_case(
query="How do I reset my password?",
expected_results="Explain the password reset process step by step.",
)
.add_case(
query="What payment methods do you accept?",
expected_results="List supported payment methods clearly.",
)
.publish()
)
print(f"Published dataset: {dataset.id}")
# 2. Publish a standalone, reusable LLM Judge Scorer, independent of this (or any) dataset's
# own bundled config, with every similarity metric turned on. Reuse this same id across other
# datasets instead of rebuilding it each time; see /sdk/evaluations/overview#reusable-grading-configs.
scorer: JudgeScorer = client.monitor.judge_scorers.builder(
"Strict - two runs",
number_of_requests=2,
acceptance_criteria="Accurate, concise, grounded in the support policy.",
rejection_criteria="No hallucinated policies or made-up steps.",
vector_similarity=True,
jaccard_similarity=True,
bleu_score=True,
rouge_score=True,
).publish()
print(f"Published judge scorer: {scorer.id}")
# 3. The agent under test, traced so each result links to a full Execution Timeline.
def support_agent(case: EvaluationCase) -> Dict[str, Any]:
with client.tracer.trace(
"support-agent-call",
input={"query": case.query},
framework="openai",
model="gpt-4o-mini",
sync=True,
monitor=False, # the run's judge already scores this case
) as span:
resp = oai.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful support agent."},
{"role": "user", "content": case.query},
],
)
span.output = resp.choices[0].message.content
return {
"output": resp.choices[0].message.content,
"metadata": {"model": resp.model},
"trace_id": span.trace_id,
}
# 4. Run: .execute() calls support_agent once per case and scores each response immediately;
# .finalize() closes the run. Both return EvaluationRunContext (self), not a Report. Only
# .analyze() returns one. Typing run_context/report as two separate variables means a type
# checker catches it immediately if .analyze() is ever skipped and something reads
# report.average_rating off the wrong type.
run_context: EvaluationRunContext = (
client.evaluations.run(
dataset_id=dataset.id,
subject={
"kind": "custom_agent",
"displayName": "GPT-4o-mini Support Agent",
"framework": "openai",
},
scorer_id=scorer.id,
)
.execute(support_agent)
.finalize()
)
# Available immediately, no .analyze() call needed just to see the score.
print(f"Average rating: {run_context.average_rating:.2f} ({run_context.rated_count} rated)")
# 5. Analyze (optional): adds the qualitative report (strengths, weaknesses, recommendations).
report: Report = run_context.analyze()
print(f"Cosine similarity: {report.cosine_similarity:.3f}" if report.cosine_similarity is not None else "")
print(f"Dashboard: {report.dashboard_url}")
```
Each result's dashboard row now has a "View trace" action opening the full Execution Timeline recorded in step 3, not just the rating.
A complete working example, with tool use and a reasoning-model (`o4-mini`) variant, is available as [`openai_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/openai_eval.py) in the AgentX-Python repository.
# OpenAI Agents SDK
Source: https://developers.agentx.so/sdk/evaluations/examples/openai-agents
Evaluate an agent built on the OpenAI Agents SDK (the `agents` / `openai-agents` package)
This is a different package from plain [`openai`](/sdk/evaluations/examples/openai): the Agents SDK (`import agents`) adds its own `Agent`/`Runner` abstraction, tool calling, and handoffs on top of the Chat Completions/Responses APIs. Use this page if your code imports from `agents`; use the OpenAI page if you're calling `openai.chat.completions.create()` directly.
Install:
```bash theme={null}
pip install agentx-python openai-agents
```
## Usage
```python theme={null}
from agentx import AgentX
from agents import Agent, Runner, function_tool
client = AgentX.from_env()
@function_tool
def get_policy(topic: str) -> str:
"""Look up a company policy by topic."""
db = {
"cancel": "Go to Account → Subscription → Cancel.",
"refund": "Full refund within 30 days.",
}
return db.get(topic.lower(), "No policy found.")
agent = Agent(
name="support-agent",
instructions="You are a helpful support agent. Use get_policy to look up policies.",
tools=[get_policy],
)
def openai_agents_eval(case):
result = Runner.run_sync(agent, case.query)
return {"output": result.final_output}
run_context = (
client.evaluations
# "openai-agents" isn't a valid EvaluationSubject.framework value (see note below), so "other"
# is the correct value here, distinct from what you pass to tracer.trace(framework=...).
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Support Agent", "framework": "other"})
.execute(openai_agents_eval)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
`EvaluationSubject.framework` only accepts a fixed set of values (`"raw_python"`, `"openai"`, `"anthropic"`, `"google"`, `"langchain"`, `"llamaindex"`, `"crewai"`, `"autogen"`, `"n8n"`, `"flowise"`, `"other"`). `"openai-agents"` isn't one of them, so `subject.framework` must be `"other"` here. This is unrelated to `tracer.trace(framework=...)` below, which accepts any string and does use `"openai-agents"`; the two `framework` fields are independent and differently constrained.
## With tracing
Unlike Anthropic/LangChain/CrewAI, there's no way to get `trace_id` back from the Agents SDK's own tracing integration ([`AgentXTracingProcessor`](/sdk/integrations/openai-agents)). It's a processor you register once at startup and it reports on its own schedule as runs complete elsewhere in the SDK, not a span your eval function controls. To link a trace to an eval result, wrap `Runner.run_sync(...)` in `tracer.trace(..., sync=True)` directly instead, the same pattern as the [raw OpenAI example](/sdk/evaluations/examples/openai#with-tracing):
```python theme={null}
from agentx import AgentX
from agents import Agent, Runner
client = AgentX.from_env()
agent = Agent(name="support-agent", instructions="You are a helpful support agent.")
def openai_agents_eval(case):
with client.tracer.trace(
"openai-agents-call",
input={"query": case.query},
framework="openai-agents",
sync=True,
monitor=False, # the run's judge already scores this case
) as span:
result = Runner.run_sync(agent, case.query)
span.output = result.final_output
return {
"output": result.final_output,
"trace_id": span.trace_id,
}
```
If you *also* want every other (non-eval) run of this agent traced automatically, not just the ones going through an evaluation, register [`AgentXTracingProcessor`](/sdk/integrations/openai-agents) globally in addition to the pattern above. The two don't conflict, but they are independent: the processor's traces are separate from, and won't be linked to, any eval result's `trace_id`.
## `EvaluationCase` fields
Same contract as every other framework: see [Examples overview](/sdk/evaluations/examples/overview) for the full `EvaluationCase`/return-value reference. `case.query` is what you pass to `Runner.run_sync(agent, case.query)`.
# Examples
Source: https://developers.agentx.so/sdk/evaluations/examples/overview
The common pattern behind every framework integration
All examples follow the same pattern: wrap your framework's output in a function that accepts an `EvaluationCase` and returns a `str`, `dict`, or `EvaluationResult`.
```python theme={null}
from agentx.evaluations.models import EvaluationCase
def my_agent(case: EvaluationCase) -> str:
return f"Answer to: {case.query}"
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "My Bot", "framework": "raw_python"})
.execute(my_agent)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}") # no .analyze() needed for this
```
`EvaluationCase` gives your function:
| Field | Type | Description |
| ------------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `case.query` | `str` | The test question's text |
| `case.case_id` | `str` | Stable id for this case, e.g. `"case-0"` |
| `case.question_index` | `int` | 0-based index into the dataset's `questions` array |
| `case.run_number` | `int` | Which repetition this is (1 through the config's `number_of_requests`) |
| `case.expected_results` | `str \| None` | The question's expected-answer text, if the dataset defines one |
| `case.expected_capabilities` / `case.expected_knowledge_base` / `case.expected_delegations` | `list[str] \| None` | The question's other expectations, when the dataset defines them |
| `case.model` | `str \| None` | Set only when Sovereignty & Portability is enabled: which comparison model this run should use |
| `case.is_smoke_test_variant` | `bool` | `True` when `case.query` is a server-generated [smoke-test paraphrase](/sdk/evaluations/build-dataset#smoke-testing) rather than the original question |
What you do inside the function is entirely up to you: call an LLM directly, invoke a LangChain chain, kick off a CrewAI crew, or hit an HTTP endpoint. What you return determines what AgentX sees:
| Return type | Behavior |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `str` | Used directly as the response text |
| `dict` | `output`/`text`/`response` for the text, plus optional `metadata`, `input_tokens`, `output_tokens`, `trace_id`, `retrieval_context` (what the agent retrieved - feeds [RAG judging](/evaluation/rag)), `error` |
| `EvaluationResult` | Full manual control: set every field yourself, including `observable_trace` |
Returning a plain `str` is enough to get scored. Add `metadata: {"model": ...}` when you want the Sovereignty & Portability breakdown, or `trace_id` when you want a full [Execution Timeline](/sdk/tracing#linking-a-trace-to-an-evaluation-result) attached to the result. Neither is required.
Pick your framework:
Plain OpenAI chat completions
Agent/Runner, tool calling, handoffs
Claude via the Anthropic SDK
Prompt templates and chains
Multi-agent crews
Conversable agents
RAG query engines
Any agent exposed as a web service
Score outputs you already generated
A complete, runnable version of each example lives in the [AgentX-Python repository](https://github.com/AgentX-ai/AgentX-Python/tree/main/examples/evaluations).
# Submit Pre-Defined Results
Source: https://developers.agentx.so/sdk/evaluations/examples/precomputed-results
Score outputs you already generated, without re-running the agent
No install needed beyond `agentx-python` itself. Use `PrecomputedAdapter` when you already have your agent's outputs (from a batch job, a low-code tool like n8n, or a previous run) and just want them scored, without invoking a live callable or HTTP endpoint.
## Usage
```python theme={null}
from agentx import AgentX
from agentx.evaluations.adapters.precomputed import PrecomputedAdapter
client = AgentX.from_env()
outputs = {
"case-0": "To reset your password, go to Login → Forgot Password.",
"case-1": {
"output": "We accept Visa, Mastercard, PayPal, and bank transfers.",
"metadata": {"source": "n8n-export", "model": "gpt-4o"},
},
}
adapter = PrecomputedAdapter(outputs)
run_context = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "n8n Batch", "framework": "n8n", "runtime": "low_code"})
.execute(adapter)
.finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")
```
### `PrecomputedAdapter(outputs)`
```python theme={null}
PrecomputedAdapter(outputs: list[Any] | dict[str, Any])
```
`outputs` is either:
* **A `dict` keyed by `case_id`** (the format shown above): `"case-{questionIndex}"` for a dataset's `N`th question, e.g. `"case-0"`, `"case-1"`. This is the recommended form.
* **A plain `list`**, positionally matched to question index: `outputs[0]` is the answer for question 0, and so on. Internally this is just sugar for `{"0": outputs[0], "1": outputs[1], ...}`.
Each value is either the output text directly (a `str`), or a `dict` for full control, using the same shape as any other adapter's return value: `output`/`text`, plus optional `metadata`, `input_tokens`, `output_tokens`, `trace_id`. Every case in the dataset must have a matching key, or it's submitted with an empty response.
`case_id` doesn't vary by repetition. If the dataset's config runs each question more than once (`number_of_requests > 1`), every repetition of a question looks up the **same** key and gets the **same** precomputed output submitted for all of them. `PrecomputedAdapter` is built for the common case of one precomputed answer per question; if you have distinct precomputed outputs per repetition, submit results directly via the REST results endpoint instead (`POST /custom-agent-evaluations/runs/:id/results`), setting a distinct `runNumber` and `idempotencyKey` per call.
A complete working example is available as [`precomputed_results_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/precomputed_results_eval.py) in the AgentX-Python repository, or [`csv_import_eval.py`](https://github.com/AgentX-ai/AgentX-Python/blob/main/examples/evaluations/csv_import_eval.py) for loading the dataset itself from a CSV file first (see [Build Dataset](/sdk/evaluations/build-dataset#csv-import)).
# Installation
Source: https://developers.agentx.so/sdk/evaluations/installation
Install the AgentX Python SDK and the framework extras you need
## Core package
```bash theme={null}
pip install agentx-python python-dotenv
```
Requires Python 3.9+.
## Framework-specific packages
Install only what your agent needs.
| Framework | Install |
| ----------------- | --------------------------------------------------------------- |
| OpenAI | `pip install openai` |
| OpenAI Agents SDK | `pip install openai-agents` |
| Anthropic | `pip install anthropic` |
| Google Gemini | `pip install google-genai` |
| LangChain | `pip install langchain langchain-openai` |
| LlamaIndex | `pip install llama-index llama-index-llms-openai` |
| CrewAI | `pip install crewai` |
| AutoGen / AG2 | `pip install pyautogen` (or `pip install ag2` for the AG2 fork) |
These are plain framework packages; evaluating an agent doesn't need any
`agentx-python[extra]` install. Those extras are only for
[tracing](/sdk/tracing) auto-instrumentation (`AgentXCallbackHandler`,
`patch_anthropic_client`, etc.), which evaluation examples don't use.
For Google, install `google-genai`, not the deprecated `google-generativeai`
package.
## Environment
```bash theme={null}
export AGENTX_API_KEY="agtx_local_a1b2c3d4e5f60718293a4b5c6d7e8f9012345678"
```
Then in code:
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
client.ping() # optional but recommended: fails fast on a wrong URL or key
```
Constructing the client never touches the network (standard SDK behavior - a bad `base_url`
or `api_key` won't error until first use), and trace delivery is fire-and-forget, so a
misconfigured client would otherwise only surface as a one-time warning in logs.
`client.ping()` makes one cheap authenticated call and raises `AgentXConnectionError` (bad
URL) or `AgentXAuthError` (rejected key) with an actionable message - call it once at startup
of anything long-running.
See [Python SDK Overview](/sdk/overview) for the full initialization options, including self-hosted `base_url` overrides.
# Offline Evaluation
Source: https://developers.agentx.so/sdk/evaluations/overview
Score any agent from any framework or provider against a dataset - LLM-judged, on demand, before or after you ship
**Offline evaluation** catches problems before your users do. It works like unit and
integration tests for agent behavior: a fixed dataset of test cases with known-good answers,
run on demand - before a release, after a prompt change, in CI - with reproducibility and
stability as the point. Because you control the cases, you have **ground truth**: every case
can carry an expected answer, the tool calls a correct run should make, even the chunks a
correct retriever should fetch, and the scoring can be as strict as an exact deterministic
check or as nuanced as an LLM judge.
In the dashboard this is the **Evaluate** tab. Its counterpart is
[Online Evaluation](/sdk/monitor) (the Monitor tab), which scores live traffic where no
ground truth exists.
**Doing it well:**
* **Test intermediate steps, not just final answers.** A case can pin the
[expected tool-call trajectory](/sdk/evaluations/build-dataset#expected-trajectories) and the
[expected retrieval context](/sdk/evaluations/build-dataset#expected-retrieval-context), so a
RAG pipeline's retriever and a ReAct agent's tool choices are graded independently of the
final text.
* **Mix deterministic and judged scoring.** [Code scorers](/evaluation/code-scorers),
similarity metrics, and trajectory/context matching are free and exactly reproducible; the
LLM judge handles what rules can't express. Low-variance checks first, judgment second.
* **Build the golden dataset from production.** The best test cases are real failures -
[one click turns a bad trace or signal into a dataset case](/evaluation/datasets-from-production),
provenance included, so every incident becomes a permanent regression test.
* **Make it a gate.** [`run.gate(fail_under=..., no_regression=True)`](/sdk/ci-cd) turns any
run into a CI pass/fail, so quality regressions block the merge instead of shipping.
The Custom Agent Evaluation SDK lets you run your agent against a dataset of test questions and get back LLM-judged scores, similarity metrics, and an AI-generated analysis report. It works with **any agent you own**: LangChain, CrewAI, AutoGen/AG2, LlamaIndex, OpenAI, Anthropic, an HTTP endpoint, or plain Python.
Unlike [CI/CD evaluation](/sdk/ci-cd), which is built for a fast pass/fail gate in a pipeline, Custom Agent Evaluations are built for depth: multi-run consistency checks, similarity scoring, and a full report with strengths, weaknesses, and recommendations.
## How it works
```
Your agent (local)
│
│ agentx-python SDK
▼
AgentX API ──► scores every response, runs similarity metrics
│
▼
Report ──► terminal output + dashboard
```
1. **Build a dataset**: create test cases with queries and (optionally) expected results.
2. **Run your agent**: the SDK calls your function or adapter once per test case; each response is scored immediately.
3. **Finalize**: closes the run. `average_rating`, `min_rating`, `max_rating`, and `rated_count` are already available at this point; no need to wait for analysis just to see how it went.
4. **Analyze (optional)**: AgentX generates the full qualitative report, covering strengths, weaknesses, instruction adherence, and recommendations.
5. **Review results**: read the report in your terminal or open the dashboard link.
## Installation
```bash theme={null}
pip install agentx-python
```
See [Installation](/sdk/evaluations/installation) for framework-specific extras.
## What's included
| Module | Description |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `client.evaluations` | `EvaluationsRunner`, builds and runs evaluations against a dataset |
| `client.evaluations.run()` | Starts a run, returns a chainable `EvaluationRunContext` |
| `client.evaluations.datasets` | Dataset builder and CSV import |
| `client.monitor.judge_scorers` | The unified LLM Judge Scorer: one rubric, offline (dataset-run) profile, and optional online profile |
| `client.evaluations.settings` | Legacy view of a scorer's offline profile, kept for existing code - see [Evaluation Settings](/sdk/evaluations/evaluation-settings) |
| `agentx.evaluations.models.EvaluationCase` | The test case passed to your agent function |
| `agentx.evaluations.models.EvaluationResult` | Optional return type for full control over a result |
| `agentx.evaluations.adapters` | `PrecomputedAdapter`, `HttpEndpointAdapter`: evaluate without a live Python callable |
### Reusable grading configs
A dataset's questions and its grader are independent, so the same LLM Judge Scorer can grade any dataset. Build one once with `client.monitor.judge_scorers.builder(...).publish()` and pass its id as `scorer_id` to `client.evaluations.run(...)`. Omit it to use the dataset's own bundled config, unchanged. Self-host runs can also pass `additional_scorer_ids=[...]` to grade every result with extra judges in the same run - see [multiple judge scorers](/sdk/evaluations/quickstart#multiple-judge-scorers-per-run). This is purely additive. (`evaluation_settings_id` and the legacy `client.evaluations.settings.builder` keep working as aliases - see [Evaluation Settings](/sdk/evaluations/evaluation-settings).)
## Guides
Framework-specific package extras
Run your first evaluation in a few lines
Create test cases programmatically or from CSV
Build a standalone, reusable grading config
Describe the agent being evaluated
What .analyze() returns: strengths, weaknesses, recommendations
Framework-by-framework integration examples
Attach a full Execution Timeline to each eval result, not just a score
Gate releases with a pass/fail threshold instead
# Quick Start
Source: https://developers.agentx.so/sdk/evaluations/quickstart
Run your first custom agent evaluation in a few lines
## Prerequisites
1. [Install the SDK](/sdk/evaluations/installation).
2. [Build a dataset](/sdk/evaluations/build-dataset), or use an existing dataset ID from the AgentX dashboard.
3. Export `AGENTX_API_KEY` in your environment.
## Run an evaluation
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
def my_agent(case):
return f"Answer to: {case.query}"
run_context = (
client.evaluations
.run(
dataset_id="existing-dataset-id",
subject={"kind": "custom_agent", "displayName": "My Agent", "framework": "raw_python"},
)
.execute(my_agent)
.finalize()
)
# average_rating/min_rating/max_rating/rated_count are already available here, no need to
# call .analyze() just to see how the run went.
print(f"Average rating: {run_context.average_rating:.2f}")
report = run_context.analyze()
print(f"Dashboard: {report.dashboard_url}")
```
`client.evaluations.run()` creates the run, `.execute()` calls `my_agent` once per test case, scores each response immediately, and submits the results. `.finalize()` closes the run: `run_context.average_rating` (and `.min_rating` / `.max_rating` / `.rated_count`) read straight from those scores, no LLM analysis pass required. `.analyze()` is a separate, optional step that adds the qualitative report (strengths, weaknesses, recommendations) and returns it as `report`.
## Link each result to its trace
Trace the agent call and return the span's id alongside the output - three things light up at
once: a **View trace** button on every result row, **trajectory-aware judging** (the judge sees
the tools the agent actually called, not just the answer), and
[expected-trajectory matching](/sdk/evaluations/build-dataset#expected-trajectories) for cases
that declare `expected_tools`:
```python theme={null}
def my_agent(case):
# sync=True populates span.trace_id before the block exits; monitor=False keeps the run's
# own judge as the only scorer (no double-judging at ingest).
with client.tracer.trace(
"my-agent", input={"query": case.query}, sync=True, monitor=False
) as span:
answer = run_my_agent(case.query)
span.output = answer
return {"output": answer, "trace_id": span.trace_id}
```
The dashboard's **Create evaluation** button (Evaluate → Runs) generates this exact scaffold
with your real dataset and evaluator ids inlined, in Simple, OpenAI, LangChain, or Google ADK
flavors.
## Concurrency, output reuse, and resume
`.execute()` takes two optional knobs:
```python theme={null}
run_context = (
client.evaluations
.run(dataset_id="existing-dataset-id", subject=subject)
.execute(my_agent, concurrency=4) # cases run in a thread pool
.finalize()
)
# Iterating on grading config? Replay the previous run's recorded outputs instead of
# calling the agent again - unchanged cases are re-scored with THIS run's grading config.
run_context = (
client.evaluations
.run(dataset_id="existing-dataset-id", subject=subject, scorer_id=stricter_scorer.id)
.execute(my_agent, reuse_outputs_from="previous-run-id")
.finalize()
)
```
`concurrency` (default `1`) bounds parallel agent calls; results are still submitted in case
order. `reuse_outputs_from` replays the recorded outputs of cases unchanged since that run and
only invokes your agent for new or edited ones - scorer iteration costs judge calls, not agent
calls. Replayed results carry `reusedFromRun` in their metadata.
Interrupted runs resume for free: the engine tracks which case results a run already received,
and re-running `.execute()` against the same run skips them (older engines without the route
just re-run everything). A batch that fails to submit is retried once and then raises
`EvaluationSubmissionError` instead of letting the run finish silently empty - the run is left
unfinalized, and re-running `execute()` picks up past the already-submitted cases.
## Configuration
`AgentX` reads environment variables via `.from_env()`:
| Variable | Description |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTX_API_KEY` | Workspace API key (required) |
| `AGENTX_API_BASE_URL` | Optional override for the API base URL |
| `AGENTX_WORKSPACE_ID` | Explicit workspace ID. Set this if your API key's user belongs to more than one workspace, otherwise datasets, configs, and runs are created in whichever workspace the key defaults to, which may not be the one you intend. |
Or pass configuration directly:
```python theme={null}
client = AgentX(
api_key="your-key",
base_url="https://your-agentx-instance.com/api/v1", # include the /api/v1 suffix
workspace_id="your-workspace-id",
)
```
## Next steps
Create test cases instead of using an existing dataset
Describe your agent so analysis can check instruction adherence
Framework-specific integration patterns
Score outputs you already generated, without re-running the agent
Build one grading config once, run it against any dataset
Get a full Execution Timeline per result, not just a score
## Multiple judge scorers per run
One agent execution, N verdicts (self-host): pass `additional_scorer_ids` and every result is
also scored by each extra judge, with its own rubric and model. The primary scorer keeps the
`rating` column (gates and averages unchanged); extra verdicts land in each result row's
`judgeScorerResults` and roll up into the run's `scorerBreakdown`.
```python theme={null}
run = client.evaluations.run(
dataset_id,
subject,
scorer_id=quality.id,
additional_scorer_ids=[safety.id, tone.id],
).execute(my_agent).finalize()
for row in run.results():
extras = {v["name"]: v["rating"] for v in (row.judge_scorer_results or [])}
print(row.rating, extras) # e.g. 9.0 {'Safety': 3.0, 'Tone': 6.0}
```
The CI gate can target a named additional scorer - "fail if Safety is low even when the
average looks fine":
```python theme={null}
gate = run.gate(fail_under=8, scorer="Safety") # gates Safety's own average
print(gate.passed, gate.gated_scorer) # e.g. False {'id': '...', 'name': 'Safety'}
```
(Equivalent REST: `GET /custom-agent-evaluations/runs/{id}/gate?failUnder=8&scorer=Safety`;
an unknown scorer id or name is a hard 400, never a silently-passing gate.)
# Anthropic
Source: https://developers.agentx.so/sdk/integrations/anthropic
Auto-trace Anthropic SDK calls with patch_anthropic_client
Install the integration extra:
```bash theme={null}
pip install "agentx-python[anthropic]"
```
## Usage
Call `patch_anthropic_client()` once after creating your Anthropic client. All subsequent `client.messages.create()` and `client.messages.stream()` calls are traced automatically. No changes to individual API calls are needed.
Works with both `anthropic.Anthropic` and `anthropic.AsyncAnthropic` - pass whichever client you use, sync or async, streaming or not.
```python theme={null}
from agentx import AgentX
from agentx.integrations.anthropic import patch_anthropic_client
import anthropic
agentx = AgentX.from_env()
client = anthropic.Anthropic()
patch_anthropic_client(
client,
tracer=agentx.tracer,
name="claude-support-agent",
metadata={"env": "production"},
session_id="session-xyz-789",
)
# Regular call, traced automatically
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
# Streaming call, also traced automatically
with client.messages.stream(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": "What is your refund policy?"}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
## What gets traced
By default, each `messages.create()` or `messages.stream()` call produces its own trace.
| Field | Source |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `messages` kwarg, with `system` prepended as a leading `{"role": "system", ...}` entry when set |
| `output` | The response's text blocks (or a tool-call description if the reply is a pure tool call), streamed final message included |
| `latencyMs` | Wall-clock time of the API call - measured after the real response comes back, for both sync and async clients |
| `model` | `model` kwarg |
| `inputTokens` / `outputTokens` | `response.usage.input_tokens` / `.output_tokens`. The input total folds in `cache_creation_input_tokens` and `cache_read_input_tokens`, so it reflects real context-window usage when prompt caching is on |
| `cacheReadTokens` / `cacheWriteTokens` | `response.usage.cache_read_input_tokens` / `.cache_creation_input_tokens` - reported as subsets of `inputTokens` so the cost chart prices them at the cache rate |
| `framework` | Stamped `anthropic` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `metadata.tools` | The request's `tools=[...]` definitions, captured for the unregistered-tool listing (Tools & MCPs) |
| `error` | Exception message if the API call raises |
The raw Anthropic SDK has no built-in concept of "tool call" or "retrieval"; those only exist as plain Python code around your `messages.create()` calls, so the patch can't see them on its own. Use `tracer.trace_tool_call()` and `tracer.trace_retrieval()` (below) to record them manually so they show up in the trace's performance summary.
## Tool use
Anthropic tool use is a manual loop: you call `messages.create()`, execute whatever tool the model requested yourself, then call `messages.create()` again with the result. The tool execution happens in plain Python between the two API calls, so wrap it in `tracer.trace_tool_call()` to record it:
```python theme={null}
from agentx import AgentX
from agentx.integrations.anthropic import patch_anthropic_client
import anthropic
agentx = AgentX.from_env()
client = anthropic.Anthropic()
patch_anthropic_client(client, tracer=agentx.tracer, name="claude-support-agent-tool")
def policy_lookup(topic: str) -> str:
"""Look up a company policy by topic."""
db = {
"cancel": "Go to Account → Subscription → Cancel.",
"trial": "14-day free trial, no credit card required.",
"refund": "Full refund within 30 days.",
}
for key, val in db.items():
if key in topic.lower():
return val
return "No policy found."
tools = [
{
"name": "policy_lookup",
"description": "Look up a company policy by topic.",
"input_schema": {
"type": "object",
"properties": {"topic": {"type": "string"}},
"required": ["topic"],
},
}
]
# Dispatch table. Don't hardcode a single tool name. Any tool the model
# calls gets looked up here and traced by its real name.
TOOL_REGISTRY = {"policy_lookup": policy_lookup}
messages = [{"role": "user", "content": "How do I cancel my subscription?"}]
# Wrap the whole loop in one span so every messages.create() call below
# collapses into a single trace instead of one trace per call, see
# "Multi-call agentic loops" below.
with agentx.tracer.trace("claude-support-agent-tool", framework="anthropic") as span:
span.input = messages[0]["content"]
while True:
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=512,
tools=tools,
messages=messages,
)
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason != "tool_use":
break
tool_results = []
for block in response.content:
if block.type != "tool_use":
continue
fn = TOOL_REGISTRY.get(block.name)
with agentx.tracer.trace_tool_call(block.name, input=block.input) as t:
result = fn(**block.input) if fn else f"Unknown tool: {block.name}"
t.output = result
tool_results.append(
{"type": "tool_result", "tool_use_id": block.id, "content": result}
)
messages.append({"role": "user", "content": tool_results})
for block in response.content:
if block.type == "text":
span.output = block.text
print(block.text)
agentx.tracer.flush(timeout=10)
```
### `tracer.trace_tool_call()`
```python theme={null}
tracer.trace_tool_call(name: str, *, input: Any = None) -> contextmanager
```
Times the block automatically. Assign `t.output` before the block exits.
| Parameter | Description |
| --------- | ------------------------------------------ |
| `name` | Tool name, shown in the trace's tool calls |
| `input` | The arguments passed to the tool |
For cases where you already have the timing and result computed, use `tracer.record_tool_call()` directly instead of the context manager:
```python theme={null}
tracer.record_tool_call(
name: str,
*,
input: Any = None,
output: Any = None,
latency_ms: int | None = None,
) -> None
```
## Retrieval-augmented generation (RAG)
Like tool calls, a hand-rolled retrieval step (vector search, keyword lookup, etc.) run before `messages.create()` is invisible to the patch. Record it with `tracer.trace_retrieval()`:
```python theme={null}
from agentx import AgentX
from agentx.integrations.anthropic import patch_anthropic_client
import anthropic
agentx = AgentX.from_env()
client = anthropic.Anthropic()
patch_anthropic_client(client, tracer=agentx.tracer, name="claude-support-agent-rag")
knowledge_base = [
"To cancel your subscription, go to Account → Subscription → Cancel.",
"We offer a 14-day free trial, no credit card required.",
"Full refunds are available within 30 days of purchase.",
]
def retrieve(query: str, k: int = 2) -> list[str]:
query_words = set(query.lower().split())
score = lambda doc: len(query_words & set(doc.lower().split()))
return sorted(knowledge_base, key=score, reverse=True)[:k]
question = "How do I cancel my subscription?"
with agentx.tracer.trace_retrieval("kb_search", query=question) as r:
docs = retrieve(question)
r.doc_count = len(docs)
r.output = docs # the retrieved chunks - what RAG judges grade against
context = "\n".join(f"- {doc}" for doc in docs)
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
system="Answer using only the provided context.",
messages=[
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
],
)
print(response.content[0].text)
agentx.tracer.flush(timeout=10)
```
### `tracer.trace_retrieval()`
```python theme={null}
tracer.trace_retrieval(name: str = "Retrieval", *, query: str | None = None) -> contextmanager
```
Times the block automatically. Assign `r.doc_count` and `r.output` inside the block.
| Parameter | Description |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Step name, shown in the trace's timeline (any name works - retrieval spans carry an explicit kind marker) |
| `query` | The retrieval query, shown in the trace |
| `r.output` | Set inside the block: the retrieved chunks. This is what [RAG judging](/evaluation/rag) reads as `{context}` - without it the retrieval is timed but contributes no chunks |
| `r.doc_count` | Set inside the block: how many documents came back |
The context manager attaches the retrieval to whichever trace is sent next: either the current active span (see below) or the next standalone `messages.create()` call.
A retrieval or tool call recorded with no active span is queued and merged
into the very next trace this tracer sends, so call `trace_retrieval()` /
`trace_tool_call()` immediately before the `messages.create()` call it
belongs to.
## Multi-call agentic loops
Tool-use loops and multi-turn agents call `messages.create()` more than once. By default each call is its own independent trace. Wrap the whole loop in `with tracer.trace(...)` to collapse it into **one** trace instead: every `messages.create()` call made while that span is active is folded in as an `"LLM Call N"` step (in call order, interleaved with any `trace_tool_call()` steps):
```python theme={null}
with agentx.tracer.trace("claude-support-agent-tool", framework="anthropic") as span:
span.input = question
# ... call client.messages.create() as many times as needed ...
span.output = final_answer
```
This works because `patch_anthropic_client()` checks `tracer.current_span` on every call: if a span is active on the current thread, the call is attached to it; otherwise it sends its own trace as usual. No other code changes are needed. The same patched client works standalone or inside a span.
Without the `with tracer.trace(...)` wrapper, a tool-use loop like the one
above still produces a working trace per call, but the tool call recorded
via `trace_tool_call()` attaches to whichever trace is sent *next*, which is
the following turn, not the turn that requested the tool. Wrapping the loop
in a span keeps everything on one trace, in the right order.
## `patch_anthropic_client()` reference
```python theme={null}
patch_anthropic_client(
client: anthropic.Anthropic | anthropic.AsyncAnthropic,
tracer: Tracer,
name: str = "anthropic-agent",
metadata: dict | None = None,
session_id: str | None = None,
) -> None
```
| Parameter | Description |
| ------------ | ----------------------------------------------------------------------------- |
| `client` | The `anthropic.Anthropic()` or `anthropic.AsyncAnthropic()` instance to patch |
| `tracer` | `agentx.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI for every trace |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
Calling `patch_anthropic_client()` on an already-patched client is a no-op; it is safe to call multiple times.
## Full example
```python theme={null}
from agentx import AgentX
from agentx.integrations.anthropic import patch_anthropic_client
import anthropic
agentx = AgentX.from_env()
client = anthropic.Anthropic()
patch_anthropic_client(
client,
tracer=agentx.tracer,
name="claude-support-agent",
metadata={"env": "production"},
session_id="session-xyz-789",
)
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
print(response.content[0].text)
agentx.tracer.flush(timeout=10)
```
# AutoGen
Source: https://developers.agentx.so/sdk/integrations/autogen
Trace Microsoft AutoGen agent and team runs with AgentXAutoGenObserver
Targets the modern `autogen-agentchat` / `autogen-core` architecture (the actively maintained v0.4+ rewrite) - not the older `pyautogen` / `ag2` fork.
Install the integration extra:
```bash theme={null}
pip install "agentx-python[autogen]"
```
## `observer.run()`
Wraps an `AssistantAgent` or `Team`'s `.run(task=...)` call. Since AutoGen's own API is async-native, `observer.run()` is async too - `await` it the same way you'd `await agent.run(...)`.
```python theme={null}
from agentx import AgentX
from agentx.integrations.autogen import AgentXAutoGenObserver
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
agentx = AgentX.from_env()
observer = AgentXAutoGenObserver(
tracer=agentx.tracer,
name="support-agent",
metadata={"env": "production"},
)
model_client = OpenAIChatCompletionClient(model="gpt-4o-mini")
agent = AssistantAgent("assistant", model_client=model_client)
result = await observer.run(agent, task="How do I cancel my subscription?")
print(result.messages[-1].content)
agentx.tracer.flush(timeout=10)
```
Works the same way for a `Team`:
```python theme={null}
result = await observer.run(team, task="Research and summarize the cancellation policy.")
```
## What gets traced
Each `.run()` call produces one trace.
| Field | Source |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `TaskResult.messages[0].content` - the task, normalized to a real message regardless of whether you passed a plain string or a message object |
| `output` | The last message's `content` |
| `latencyMs` | Wall-clock time of the full `.run()` call |
| `performanceSummary` | One execution step per message with `models_usage` set (a real LLM call produced it), **named by the speaking agent** (`message.source`) so a multi-agent team reads as its agent-turn trajectory, plus one tool-call step per matched `ToolCallRequestEvent`/`ToolCallExecutionEvent` pair |
| `inputTokens` / `outputTokens` | Summed from each message's `models_usage.prompt_tokens` / `.completion_tokens` |
| `framework` | Stamped `autogen` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `error` | Exception message if `.run()` raises |
Per-step timing is derived from each message's real `created_at` timestamp,
chained against the previous message's timestamp. AutoGen's message schema
has no explicit start/end pair per LLM call, so this is real but
approximate - the same caveat CrewAI's per-task timing has.
`observer.run()` covers `.run()` only, not `.run_stream()` - a deliberate
scope boundary, the same as this SDK's OpenAI and Google Gen AI streaming
coverage.
## `AgentXAutoGenObserver` reference
```python theme={null}
AgentXAutoGenObserver(
tracer: Tracer,
name: str = "autogen-agent",
metadata: dict | None = None,
session_id: str | None = None,
)
```
| Parameter | Description |
| ------------ | ------------------------------------------------- |
| `tracer` | `agentx.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
### `observer.run()` parameters
```python theme={null}
observer.run(
agent_or_team: AssistantAgent | Team,
task: str | BaseChatMessage | Sequence[BaseChatMessage] | None = None,
**kwargs,
) -> TaskResult
```
Any additional keyword arguments are passed straight through to `agent_or_team.run(...)`.
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# CrewAI
Source: https://developers.agentx.so/sdk/integrations/crewai
Trace CrewAI crew runs with AgentXCrewObserver
Install the integration extra:
```bash theme={null}
pip install "agentx-python[crewai]" crewai
```
## `observer.kickoff()`
Wraps `crew.kickoff()` for you. Task outputs are automatically captured as tool calls.
```python theme={null}
from agentx import AgentX
from agentx.integrations.crewai import AgentXCrewObserver
from crewai import Agent, Task, Crew
client = AgentX.from_env() # AGENTX_API_KEY (+ AGENTX_API_BASE_URL for self-host)
observer = AgentXCrewObserver(
tracer=client.tracer,
name="support-crew",
metadata={"env": "production"},
)
researcher = Agent(
role="Policy Researcher",
goal="Find relevant company policies",
backstory="You look up policies from the knowledge base.",
verbose=False,
)
writer = Agent(
role="Support Writer",
goal="Write clear, friendly support responses",
backstory="You turn policy details into helpful customer replies.",
verbose=False,
)
crew = Crew(
agents=[researcher, writer],
tasks=[
Task(
description="Look up the cancellation policy and summarize it.",
expected_output="A concise policy statement.",
agent=researcher,
),
Task(
description="Write a friendly support reply about cancelling a subscription.",
expected_output="A clear, helpful response to send to the customer.",
agent=writer,
),
],
verbose=False,
)
result = observer.kickoff(crew, inputs={"query": "cancel subscription"})
print(result.raw)
client.tracer.flush(timeout=10)
```
## What gets traced
| Field | Source |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `inputs` dict passed to `kickoff()` |
| `output` | `result.raw` (CrewOutput string) |
| `latencyMs` | Wall-clock time of the full crew run |
| `toolCalls` | Each task output, with `description` as `name` and `raw` as `output` |
| `performanceSummary` | One execution step per task, with real start/end timestamps from CrewAI's event bus |
| `framework` | Stamped `crewai` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `error` | Exception message if `crew.kickoff()` raises, or a task-level error from `TaskFailedEvent` |
Token counts are not captured by this integration - the task-level events it listens to don't
carry per-call usage.
Per-task timing comes from CrewAI's event bus (`TaskStartedEvent` / `TaskCompletedEvent` / `TaskFailedEvent`, available on modern CrewAI versions) - `observer.kickoff()` registers temporary listeners for the duration of the call, keyed by each task's real id so concurrent tasks (`async_execution=True`) are timed correctly rather than attributed to whichever task started most recently.
On older CrewAI versions that predate the event bus, `kickoff()` falls back
to splitting the total latency evenly across tasks instead - still useful
for task order, just not exact per-task duration on those versions.
## `observer.observe()` context manager
Use this instead of `kickoff()` when you need to call `crew.kickoff()` yourself, for example to catch exceptions before they reach AgentX, or to run additional logic between kickoff and reading the result:
```python theme={null}
from agentx import AgentX
from agentx.integrations.crewai import AgentXCrewObserver
from crewai import Agent, Task, Crew
client = AgentX.from_env()
observer = AgentXCrewObserver(tracer=client.tracer, name="support-crew")
researcher = Agent(
role="Policy Researcher",
goal="Find relevant company policies",
backstory="You look up policies from the knowledge base.",
)
crew = Crew(
agents=[researcher],
tasks=[
Task(
description="Look up the cancellation policy and summarize it.",
expected_output="A concise policy statement.",
agent=researcher,
)
],
)
with observer.observe(name="support-crew", input={"query": "cancel subscription"}) as span:
try:
result = crew.kickoff(inputs={"query": "cancel subscription"})
span.output = result.raw
except Exception as e:
span.set_error(str(e))
raise
client.tracer.flush(timeout=10)
```
Unlike `kickoff()`, the `observe()` context manager doesn't auto-populate `toolCalls` or `performanceSummary` from task outputs. Record them yourself with `span.add_tool_call(...)` if you need that detail.
## `AgentXCrewObserver` reference
```python theme={null}
AgentXCrewObserver(
tracer: Tracer,
name: str = "crewai-crew",
metadata: dict | None = None,
session_id: str | None = None,
)
```
| Parameter | Description |
| ------------ | ------------------------------------------------- |
| `tracer` | `client.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
### `observe()` parameters
```python theme={null}
observer.observe(
name: str | None = None,
input: Any = None,
metadata: dict | None = None,
session_id: str | None = None,
sync: bool = False,
) -> contextmanager
```
Inside the `with` block, assign `span.output` before exiting to set the trace output. Pass
`sync=True` to send synchronously so `span.trace_id` is populated once the block exits - e.g.
to [link the trace to an evaluation result](/sdk/tracing#linking-a-trace-to-an-evaluation-result).
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# Databricks
Source: https://developers.agentx.so/sdk/integrations/databricks
Trace and evaluate agents built on Databricks (Agent Bricks / Mosaic AI Agent Framework) via MLflow Tracing
Agents built on Databricks - declaratively with **Agent Bricks** or code-first with the
**Mosaic AI Agent Framework** - are auto-instrumented by **MLflow 3 Tracing**. AgentX plugs into
that in three complementary ways; pick per environment, they compose freely:
| Path | When | How |
| ------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------- |
| Push (live) | Notebooks, jobs, Model Serving endpoints that allow egress | MLflow's native OTLP export pointed at AgentX's OTel endpoint |
| Pull (batch) | Endpoints without egress config, backfills, Unity Catalog trace stores | `agentx-databricks sync` importer |
| Offline evals | Release gates, regression suites | AgentX dataset runs invoking the serving endpoint |
## Push: live OTLP export
MLflow 3 exports traces over OpenTelemetry natively. One helper sets the environment variables -
call it **before the first trace starts**:
```python theme={null}
import os
from agentx.integrations.databricks import enable_mlflow_export
enable_mlflow_export(
api_key=os.environ["AGENTX_API_KEY"],
base_url="http://localhost:4700/api/v1", # your AgentX engine
service_name="my-databricks-agent",
)
# then trace as usual - @mlflow.trace, autolog, or the Agent Framework
```
* `dual=True` (default) also keeps MLflow's own export, so the Databricks MLflow UI and
inference tables keep working (it sets `MLFLOW_TRACE_ENABLE_OTLP_DUAL_EXPORT`).
* AgentX maps MLflow's native span attributes (`mlflow.spanInputs`/`spanOutputs`/`spanType`)
directly - inputs, outputs, tool calls, and the full span tree arrive without any flags.
`genai_semconv=True` switches to OTel GenAI semantic conventions if you prefer them.
* **Deployed endpoints**: `enable_mlflow_export(dry_run=True, ...)` returns the exact variables
to paste into the Model Serving endpoint's environment-variable configuration.
## Pull: `agentx-databricks sync`
For serving endpoints where egress env vars are awkward, or to backfill history, the SDK
installs a cron-friendly importer that reads finished traces from the MLflow tracking server
(`pip install "agentx-python[databricks]"`):
```bash theme={null}
export AGENTX_API_KEY=agtx_local_...
export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
export DATABRICKS_HOST=... DATABRICKS_TOKEN=... # or MLFLOW_TRACKING_URI
agentx-databricks sync --experiment-id 123456 --since 7d # first backfill
agentx-databricks sync --experiment-id 123456 # cron this - incremental cursor
```
What each MLflow trace becomes:
| MLflow | Becomes in AgentX |
| ------------------------------- | ----------------------------------------------------------------------- |
| Trace | Root trace (deterministic `span_id` - re-syncing never duplicates) |
| Spans | Child spans with real timings - the trace dialog's Timeline/Graph views |
| `TOOL` spans | `tool_calls` on the root - tool-failure checks and trajectory matching |
| `mlflow.trace.session` metadata | Session (`dbx_`) - Sessions view, session judges |
Imported traces are stamped with the `databricks` [framework label](/trace/platform-detection).
On the push path the label comes from the OTel signal instead (`gen_ai.provider.name`, then
`gen_ai.system`, then the instrumentation scope name, then `service.name`) - for plain
`@mlflow.trace` traffic that usually means the `service_name` you passed to
`enable_mlflow_export()`.
Flags mirror `agentx-moveworks`: `--monitor` opts imported traces into ingest-time checks
(patterns, PII, tool failure), `--judge-sessions` judges each imported session afterwards
(`ifStale` - never duplicates the engine's own sweep), `--dry-run` prints payloads without
ingesting, `--agent-name` pins every trace to one agent.
## Offline evaluation of a served agent
The agent stays deployed; an AgentX dataset run drives it - see
`sample-scripts/sdk_eval_samples/databricks_agent_eval.py` for the full version:
```python theme={null}
def run_agent(case):
answer = invoke_databricks_agent(case.query) # POST /serving-endpoints//invocations
return {"output": answer}
run = client.evaluations.run(dataset_id=dataset.id, subject={...}).execute(run_agent).finalize()
```
Cases with `expected_tools=[...]` are trajectory-matched against the tools the deployed agent
actually called, once traces are linked via either path above.
# Google
Source: https://developers.agentx.so/sdk/integrations/google
Trace Google ADK agents and Gemini API calls
Google provides two SDKs relevant for tracing. Choose based on what you're building:
| SDK | Use case | Integration |
| ----------------- | --------------------------------------------------- | -------------------- |
| **Google ADK** | Full agent framework (multi-agent, tools, sessions) | `AgentXADKPlugin` |
| **Google Gen AI** | Raw Gemini API calls (`generate_content`) | `patch_genai_client` |
***
## Google ADK
Install:
```bash theme={null}
pip install "agentx-python[google-adk]" google-adk
```
Register `AgentXADKPlugin` in the `plugins` list when constructing the ADK `Runner`. Every subsequent `runner.run()` call is traced automatically.
```python theme={null}
import os
from agentx import AgentX
from agentx.integrations.google_adk import AgentXADKPlugin
from google.adk.agents import Agent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.adk.tools import FunctionTool
from google.genai import types
os.environ["GEMINI_API_KEY"] = "xxxxxxxxxxxxxxxxxxxx"
client = AgentX.from_env() # AGENTX_API_KEY (+ AGENTX_API_BASE_URL for self-host)
def get_policy(topic: str) -> dict:
"""Return the company policy for a given topic."""
db = {
"cancel": "Go to Account → Subscription → Cancel.",
"trial": "14-day free trial, no credit card required.",
"refund": "Full refund within 30 days.",
}
return {"result": db.get(topic.lower(), "No policy found.")}
agent = Agent(
name="support_agent",
instruction="You are a helpful support agent. Use get_policy to look up policies.",
tools=[FunctionTool(func=get_policy)],
)
runner = Runner(
agent=agent,
app_name="support-app",
session_service=InMemorySessionService(),
plugins=[
AgentXADKPlugin(
tracer=client.tracer,
name="support-agent",
metadata={"env": "production"},
)
],
)
session = runner.session_service.create_session_sync(
app_name="support-app", user_id="user-1"
)
for event in runner.run(
user_id="user-1",
session_id=session.id,
new_message=types.Content(
role="user", parts=[types.Part(text="How do I cancel my subscription?")]
),
):
if event.is_final_response():
print(event.content.parts[0].text)
client.tracer.flush(timeout=10)
```
### What gets traced
| Field | Source |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `input` | User message text from `on_user_message_callback` |
| `output` | Last model reply text from `after_model_callback` |
| `latencyMs` | Wall-clock time from `before_run_callback` to `after_run_callback` |
| `model` | `llm_request.model` from `before_model_callback` |
| `toolCalls` | Each `after_tool_callback`: `name`, `input`, `output`, `latencyMs` |
| `inputTokens` / `outputTokens` | Summed from each model call's `usage_metadata` (`prompt_token_count` / `candidates_token_count`) |
| `framework` | Stamped `google-adk` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `error` | `on_model_error_callback` (a failed model call) or `on_tool_error_callback` (a failed tool call) |
Each model call and tool call also becomes its own real child span with per-call timing, so the
trace dialog's Timeline shows the invocation's actual step sequence.
### `AgentXADKPlugin` reference
```python theme={null}
AgentXADKPlugin(
tracer: Tracer,
name: str = "google-adk-agent",
metadata: dict | None = None,
session_id: str | None = None,
)
```
| Parameter | Description |
| ------------ | ------------------------------------------------- |
| `tracer` | `client.tracer` from your `AgentX` instance |
| `name` | Fallback label when the agent has no name set |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
***
## Google Gen AI (raw Gemini API)
Install:
```bash theme={null}
pip install "agentx-python[google-genai]" google-genai
```
Call `patch_genai_client()` once after creating your `genai.Client`. All subsequent `client.models.generate_content()` calls are traced automatically.
Works with the async client too - pass `client.aio` instead of `client` to trace `await client.aio.models.generate_content(...)` and `async for chunk in await client.aio.models.generate_content_stream(...)` calls.
```python theme={null}
import os
from agentx import AgentX
from agentx.integrations.google_genai import patch_genai_client
from google import genai
from google.genai import types
client = AgentX.from_env() # AGENTX_API_KEY (+ AGENTX_API_BASE_URL for self-host)
genai_client = genai.Client(api_key="GEMINI_API_KEY")
patch_genai_client(
genai_client,
tracer=client.tracer,
name="gemini-support-agent",
metadata={"env": "production"},
)
# Regular call
response = genai_client.models.generate_content(
model="gemini-3.5-flash",
contents="How do I cancel my subscription?",
)
print(response.text)
# or Streaming call
for chunk in genai_client.models.generate_content_stream(
model="gemini-3.5-flash",
contents="What is your refund policy?",
):
print(chunk.text, end="", flush=True)
client.tracer.flush(timeout=10)
```
To trace the async client, patch `genai_client.aio` instead of `genai_client`:
```python theme={null}
patch_genai_client(
genai_client.aio,
tracer=client.tracer,
name="gemini-support-agent",
)
response = await genai_client.aio.models.generate_content(
model="gemini-3.5-flash",
contents="How do I cancel my subscription?",
)
print(response.text)
async for chunk in await genai_client.aio.models.generate_content_stream(
model="gemini-3.5-flash",
contents="What is your refund policy?",
):
print(chunk.text, end="", flush=True)
```
### What gets traced
By default, each `generate_content()` / `generate_content_stream()` call produces its own trace.
Wrap a multi-call loop in `with tracer.trace(...)` to collapse the calls into one trace instead,
exactly as in the [Anthropic integration](/sdk/integrations/anthropic#multi-call-agentic-loops).
| Field | Source |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `contents` argument |
| `output` | `response.text` (or a function-call description if the reply is a pure tool call); for streams, the concatenated chunk text |
| `latencyMs` | Wall-clock time of the API call (for streams, until the stream is fully consumed) |
| `model` | `model` argument |
| `inputTokens` / `outputTokens` | `usage_metadata.prompt_token_count` / `.candidates_token_count` |
| `cacheReadTokens` | `usage_metadata.cached_content_token_count` - the cached subset of `inputTokens`, priced at the cache rate |
| `framework` | Stamped `google-genai` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `metadata.tools` | The request config's `tools` function declarations, captured for the unregistered-tool listing (Tools & MCPs) |
| `error` | Exception message if the API call raises |
### `patch_genai_client()` reference
```python theme={null}
patch_genai_client(
client: genai.Client,
tracer: Tracer,
name: str = "gemini-agent",
metadata: dict | None = None,
session_id: str | None = None,
) -> None
```
| Parameter | Description |
| ------------ | ------------------------------------------------- |
| `client` | The `genai.Client()` instance to patch |
| `tracer` | `client.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI for every trace |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
Calling `patch_genai_client()` on an already-patched client is a no-op.
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# LangChain
Source: https://developers.agentx.so/sdk/integrations/langchain
Auto-trace every LangChain chain and agent run with AgentXCallbackHandler
Install the integration extra:
```bash theme={null}
pip install "agentx-python[langchain]" langchain langchain-openai
```
## LangChain Agent Executor
```python theme={null}
from agentx import AgentX
from agentx.integrations.langchain import AgentXCallbackHandler
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langchain.agents import create_agent
client = AgentX.from_env() # AGENTX_API_KEY (+ AGENTX_API_BASE_URL for self-host)
handler = AgentXCallbackHandler(
tracer=client.tracer,
name="support-agent", # custom name for the agent
session_id="session-001", # custom session id for the agent
)
@tool
def policy_lookup(topic: str) -> str:
"""Look up a company policy by topic."""
db = {
"cancel": "Go to Account → Subscription → Cancel.",
"trial": "14-day free trial, no credit card required.",
"refund": "Full refund within 30 days.",
}
for key, val in db.items():
if key in topic.lower():
return val
return "No policy found."
llm = ChatOpenAI(
model="gpt-4o-mini",
temperature=0,
api_key="sk-xxxxxxxxxxxxxxxx",
)
agent = create_agent(
llm,
tools=[policy_lookup],
system_prompt="You are a helpful support agent.",
)
result = agent.invoke(
{"messages": [{"role": "user", "content": "How do I cancel my subscription?"}]},
config={"callbacks": [handler]},
)
print(result["messages"][-1].content)
client.tracer.flush(timeout=10)
```
## LangChain Expression Language
```python theme={null}
from agentx import AgentX
from agentx.integrations.langchain import AgentXCallbackHandler
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
client = AgentX.from_env()
handler = AgentXCallbackHandler(
tracer=client.tracer,
name="support-chain",
session_id="session-001",
)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful support agent."),
("human", "{question}"),
])
chain = prompt | llm | StrOutputParser()
result = chain.invoke(
{"question": "How do I cancel my subscription?"},
config={"callbacks": [handler]},
)
print(result)
client.tracer.flush(timeout=10)
```
One `AgentXCallbackHandler` instance can be reused across multiple invocations. Pass it via LangChain's standard `config={"callbacks": [...]}`; no changes to the chain or agent definition are required.
## LangGraph
The same handler covers LangGraph (`create_agent` / `create_react_agent`) - pass it in the
invoke config exactly as above. LangGraph runs arrive as a real **span tree**: each graph node
becomes a child span, and every LLM call, tool call, and retrieval is parented under the node
that ran it, so the trace dialog's Timeline and Graph views show the actual graph trajectory.
Plumbing runnables (`RunnableSequence`, `ChannelWrite`, ...) are filtered out automatically.
```python theme={null}
agent = create_agent(ChatOpenAI(model="gpt-4o-mini"), tools=[lookup_order])
agent.invoke(
{"messages": [("user", "Where is order A-1001?")]},
config={"callbacks": [handler]},
)
```
## What gets traced
Each **top-level chain invocation** produces one trace - a root span plus child spans for graph
nodes (LangGraph), LLM calls, tool calls, and retrievals, with real per-step timings. Tool calls
are additionally mirrored onto the root's flat `toolCalls` list, which the Tool-failure check
and [trajectory matching](/concepts#trajectory-match) read.
| Field | Source |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `inputs` dict from `on_chain_start` |
| `output` | `outputs` dict from `on_chain_end` |
| `latencyMs` | Wall-clock time from chain start to end |
| `model` | Extracted from the first LLM call's serialized metadata |
| `toolCalls` | Each `on_tool_start` / `on_tool_end` pair, with `name`, `input`, `output`, `latencyMs` |
| `inputTokens` / `outputTokens` | Summed from each LLM call's `LLMResult` usage (checked under `token_usage`, `usage`, and per-generation info, so OpenAI, Anthropic, and Gemini chat models all report) |
| `framework` | Stamped `langchain` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `error` | Exception message if `on_chain_error` fires |
| `knowledgeRetrievals` | Each `on_retriever_start` / `on_retriever_end` pair - a failed retrieval (`on_retriever_error`) is recorded too, as a step with `output: "ERROR: ..."`, instead of being dropped |
Each child span states its [span kind](/trace/span-kinds): LLM calls as `llm`, tool executions
as `tool`, retriever runs as `retrieval`, LangGraph nodes as `chain`.
## `AgentXCallbackHandler` reference
```python theme={null}
AgentXCallbackHandler(
tracer: Tracer,
name: str = "langchain-agent",
metadata: dict | None = None,
session_id: str | None = None,
max_run_age_seconds: float = 3600.0,
)
```
| Parameter | Description |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tracer` | `client.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI for every trace sent by this handler |
| `metadata` | Static key-value metadata attached to every trace (max 16 KB) |
| `session_id` | Links traces from the same conversation thread in the UI |
| `max_run_age_seconds` | Housekeeping bound: run state older than this is discarded, so a callback that never fires its end event can't leak memory in a long-running server |
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# LiteLLM
Source: https://developers.agentx.so/sdk/integrations/litellm
Auto-trace every LiteLLM completion call with AgentXLiteLLMLogger
Install the integration extra:
```bash theme={null}
pip install "agentx-python[litellm]"
```
## Usage
Register `AgentXLiteLLMLogger` via `litellm.callbacks` once at startup. Every subsequent `litellm.completion()` / `litellm.acompletion()` call - sync, async, or streaming, across any of the 100+ providers LiteLLM supports - is traced automatically with no per-call changes.
```python theme={null}
from agentx import AgentX
from agentx.integrations.litellm import AgentXLiteLLMLogger
import litellm
agentx = AgentX.from_env()
litellm.callbacks = [
AgentXLiteLLMLogger(
tracer=agentx.tracer,
name="support-agent",
metadata={"env": "production"},
)
]
response = litellm.completion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
print(response.choices[0].message.content)
agentx.tracer.flush(timeout=10)
```
Async and streaming calls work the same way, no extra setup:
```python theme={null}
response = await litellm.acompletion(
model="claude-haiku-4-5-20251001",
messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
for chunk in litellm.completion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What is your refund policy?"}],
stream=True,
):
print(chunk.choices[0].delta.content or "", end="", flush=True)
```
`litellm.callbacks` is process-global - set it once at startup, not per
request. It affects every LiteLLM call made afterward, regardless of which
provider or model each call targets.
## What gets traced
By default, each `completion()` / `acompletion()` call produces its own trace. For a streamed call, LiteLLM reassembles the full response internally before invoking the logger, so streaming is traced the same way as a regular call - one trace with the complete output, not one per chunk.
| Field | Source |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `messages` kwarg |
| `output` | The choices' message content (or a tool-call description if the reply is a pure tool call) |
| `latencyMs` | `end_time - start_time` from LiteLLM's own logging callback |
| `model` | `model` kwarg |
| `inputTokens` / `outputTokens` | `response.usage.prompt_tokens` / `.completion_tokens` |
| `cacheReadTokens` | `response.usage.prompt_tokens_details.cached_tokens` - LiteLLM normalizes every provider to an OpenAI-shaped response, so this works regardless of which provider served the call |
| `framework` | Stamped `litellm` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `metadata.tools` | The request's `tools=[...]` definitions, captured for the unregistered-tool listing (Tools & MCPs) |
| `error` | The underlying exception LiteLLM captured, from `kwargs["exception"]` |
The raw LiteLLM client has no built-in concept of "tool call" or "retrieval"; those only exist as plain Python code around your `completion()` calls, so the logger can't see them on its own. Use `tracer.trace_tool_call()` and `tracer.trace_retrieval()` to record them manually so they show up in the trace's performance summary - see the [Anthropic integration's tool-use example](/sdk/integrations/anthropic#tool-use) for the same pattern.
## Multi-call agentic loops
Wrap a multi-call loop in `with tracer.trace(...)` to collapse every `completion()`/`acompletion()` call made inside it into **one** trace instead of one trace per call:
```python theme={null}
with agentx.tracer.trace("support-agent", framework="litellm") as span:
span.input = question
# ... call litellm.completion() as many times as needed ...
span.output = final_answer
```
This works because `AgentXLiteLLMLogger` checks `tracer.current_span` on every callback: if a span is active on the current thread, the call is attached to it as an `"LLM Call N"` step; otherwise it sends its own trace as usual.
## `AgentXLiteLLMLogger` reference
```python theme={null}
AgentXLiteLLMLogger(
tracer: Tracer,
name: str = "litellm-agent",
metadata: dict | None = None,
session_id: str | None = None,
)
```
| Parameter | Description |
| ------------ | ------------------------------------------------- |
| `tracer` | `agentx.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI for every trace |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
`AgentXLiteLLMLogger` is a real `litellm.integrations.custom_logger.CustomLogger` - it can be combined with other LiteLLM callbacks in the same `litellm.callbacks` list without conflict.
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# LlamaIndex
Source: https://developers.agentx.so/sdk/integrations/llamaindex
Auto-trace query engines, chat engines, and agents with AgentXLlamaIndexHandler
Install the integration extra:
```bash theme={null}
pip install "agentx-python[llamaindex]" llama-index-core
```
## Usage
Register `AgentXLlamaIndexHandler` on LlamaIndex's global `Settings.callback_manager` (or scope it to a single query engine / agent). Every subsequent top-level `query()` / `chat()` / `retrieve()` call - including its nested retrieval and LLM steps - is traced automatically.
```python theme={null}
from agentx import AgentX
from agentx.integrations.llamaindex import AgentXLlamaIndexHandler
from llama_index.core import Settings, VectorStoreIndex, Document
from llama_index.core.callbacks import CallbackManager
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
agentx = AgentX.from_env()
handler = AgentXLlamaIndexHandler(
tracer=agentx.tracer,
name="support-rag-agent",
metadata={"env": "production"},
)
Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding()
Settings.callback_manager = CallbackManager([handler])
index = VectorStoreIndex.from_documents([
Document(text="To cancel your subscription, go to Account → Subscription → Cancel."),
Document(text="We offer a 14-day free trial, no credit card required."),
])
query_engine = index.as_query_engine()
response = query_engine.query("How do I cancel my subscription?")
print(response)
agentx.tracer.flush(timeout=10)
```
To scope tracing to one query engine instead of every LlamaIndex call in the process, pass the callback manager directly instead of setting it on the global `Settings`:
```python theme={null}
query_engine = index.as_query_engine(callback_manager=CallbackManager([handler]))
```
Building the index itself (`VectorStoreIndex.from_documents(...)`) is not
traced - only real query/chat/retrieve/agent-step calls produce a trace.
Node parsing, chunking, and embedding events fired during index construction
are intentionally not sent to AgentX.
## What gets traced
Each top-level call produces one trace: a `query()`/`chat()` call, an agent step, or - if you call a retriever or LLM directly with no query engine wrapping it - that bare call itself.
| Field | Source |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | The query string, or the first LLM call's prompt for a bare `llm.complete()`/`chat()` call |
| `output` | The final `Response`/completion text |
| `latencyMs` | Wall-clock time from the top-level event's start to its end |
| `model` | `EventPayload.MODEL_NAME` from the LLM event, when the LLM integration populates it |
| `performanceSummary` | One execution step per nested `LLM` event, one retrieval step per nested `RETRIEVE` event (with `query`, `doc_count`, and the retrieved text), one tool-call step per `FUNCTION_CALL` event |
| `inputTokens` / `outputTokens` | Best-effort, summed from each LLM event's provider-raw `usage` on the completion object - LlamaIndex's callback payloads carry no dedicated token fields, so availability depends on the LLM integration |
| `framework` | Stamped `llamaindex` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `error` | Exception message from any nested event's `EventPayload.EXCEPTION` |
LlamaIndex's `CallbackManager.start_trace(trace_id)` reuses a fixed
operation-name string (`"query"`, `"chat"`, …) rather than a unique id per
call, so `AgentXLlamaIndexHandler` doesn't key state on it - it walks the
real `event_id`/`parent_id` chain instead, which stays correct under
concurrent calls in the same process.
## `AgentXLlamaIndexHandler` reference
```python theme={null}
AgentXLlamaIndexHandler(
tracer: Tracer,
name: str = "llamaindex-agent",
metadata: dict | None = None,
session_id: str | None = None,
)
```
| Parameter | Description |
| ------------ | ----------------------------------------------------------------- |
| `tracer` | `agentx.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI for every trace sent by this handler |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread in the UI |
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# Moveworks
Source: https://developers.agentx.so/sdk/integrations/moveworks
Import Moveworks AI Assistant activity into AgentX via the Data API
Moveworks agents run inside the Moveworks cloud (Agent Studio + Reasoning Engine), so there is no
in-process callback to hook the way LangChain or CrewAI integrations do - Moveworks Python Script
Actions have outbound network access disabled at the infrastructure level. Instead, this
integration **pulls activity out through the Moveworks Data API** (the read-only OData export at
`https://api.moveworks.ai/export/v1`) and replays it into AgentX:
| Moveworks record | Becomes in AgentX |
| ---------------- | ------------------------------------------------------------------------ |
| Conversation | Session (`mw_`) - Sessions view, session judges, topics |
| Interaction | Trace - input/output, timestamps, latency where derivable |
| Plugin call | `tool_calls` entry on its interaction's trace |
| Domain | Agent (`moveworks-it`, `moveworks-hr`, ...) unless you pin one name |
Every imported trace is stamped with the `moveworks` framework label, so Moveworks traffic gets
its own series in Monitor's Platforms chart and the Live Traces
[framework filter](/trace/platform-detection).
No extra dependency is needed - the importer uses only the SDK's core requirements.
## Prerequisites
* Data API credentials, minted by a Moveworks **superadmin** (Moveworks Setup → Credentials).
API key and OAuth2 client-credentials are both offered there; the importer takes the key and
sends it as `Authorization: Bearer ` (override the header/scheme if your deployment
differs).
* An AgentX project API key (self-host: Settings → API access).
## CLI: scheduled sync
The SDK installs an `agentx-moveworks` command built for cron. It keeps an incremental cursor
(`~/.agentx/moveworks_cursor.json` by default), so each run picks up where the last one ended:
```bash theme={null}
export MOVEWORKS_API_KEY=... # Data API credential
export AGENTX_API_KEY=agtx_local_... # AgentX project key
export AGENTX_API_BASE_URL=http://localhost:4700/api/v1 # self-host engine
agentx-moveworks sync --since 7d # first backfill (Data API retains 30 days)
agentx-moveworks sync # cron this - continues from the cursor
```
Useful flags: `--dry-run` prints the mapped payloads without ingesting, `--agent-name` attributes
every trace to one agent instead of per-domain, `--timestamp-field` overrides the record
timestamp field the OData `$filter` uses (default `created_time`), and `--no-cursor` ignores the
cursor file. On any ingest failure the cursor is **not** advanced, so a re-run retries the window.
Three flags control evaluation of the imported traffic:
* Pattern/built-in checks (PII, empty response, tool failure, active patterns) and trace-scoped
**online evaluators run on every imported trace automatically**, same as any live traffic -
no flag needed (`--monitor` is kept for compatibility).
* `--judge-sessions` judges every imported session with each enabled session-scoped evaluator
(Session Baseline Judge included) after the sync. Needed for backfills: the engine's automatic
session sweep only looks at the last 24 hours of activity, so older imported conversations
would otherwise never get a session verdict. The requests carry `ifStale=true`, so a session
the sweep (or a previous run) already scored is skipped, never judged twice. Each judgment is
a real LLM call - budget accordingly on large backfills.
* `--evaluate-against ` grades each **new** imported interaction's
recorded input/output against that grading config's criteria - see
[Evaluating Moveworks agents](#evaluating-moveworks-agents) below.
## Python API
```python theme={null}
from datetime import datetime, timedelta, timezone
from agentx.integrations.moveworks import MoveworksDataAPIClient, MoveworksImporter
client = MoveworksDataAPIClient(api_key="mw-data-api-key")
importer = MoveworksImporter(
client,
agentx_api_key="agtx_local_...",
agentx_base_url="http://localhost:4700/api/v1",
)
report = importer.sync(since=datetime.now(timezone.utc) - timedelta(days=1))
print(report) # conversations/interactions/ingested/failed counts
```
## Evaluating Moveworks agents
Moveworks agents run inside the Moveworks cloud and can't be invoked from outside (no inbound
converse API; Script Actions have outbound network disabled). That constrains *how* each kind
of evaluation applies - but both kinds work:
**Online evaluation - fully supported.** Every synced interaction rides the normal ingest
pipeline: trace-scoped online evaluators sample and score it, patterns raise signals, and
`--judge-sessions` gets whole conversations judged (goal progression, consistency,
resolution). One caveat to hold honestly: the Data API has a \~24h freshness SLA, so this is
*continuous retroactive* scoring - yesterday's production judged today - not
at-the-moment scoring. Signals, triage, calibration, and judge tuning all apply unchanged.
**Offline evaluation - grade recorded behavior instead of re-running the agent.** A classic
dataset run calls your agent per case; with no way to invoke a Moveworks agent, the offline
paths work from what the agent already did:
* `--evaluate-against ` grades each imported interaction against a grading config during
the sync (one judge call per **new** interaction - the engine's span dedupe means re-syncing
a window never re-bills). Each verdict is recorded as a real one-result evaluation run, so
ratings and justifications show up under Evaluate → Runs, and the report prints the average:
```bash theme={null}
agentx-moveworks sync --since 7d --evaluate-against
# MoveworksSyncReport(..., traces_evaluated=142 (avg 7.3/10), eval_skipped_deduped=0, ...)
```
* The same thing per trace from code: `client.tracer.evaluate_trace(trace_id, dataset_id)`.
* **Datasets from production**: any synced trace or conversation converts to a golden dataset
case ([multi-turn included](/evaluation/datasets-from-production)) - the regression suite for
grading future synced traffic, or the benchmark for evaluating a candidate replacement.
* **Model portability**: replay a synced interaction's input against other models
(`client.monitor.run_model_portability(trace_id, [...])`) for a cost/quality comparison.
* If your deployment exposes any inbound HTTP entry point, the
[`http_endpoint` adapter](/sdk/evaluations/examples/http-endpoint) closes the loop with real
dataset runs.
## Behavior and limits
* **Idempotent**: every trace carries a deterministic `span_id` (`mw:`) and the
engine skips spans it has already ingested, so re-syncing a window never duplicates.
* **Historical timestamps are honored**: traces land in AgentX dated when the conversation
happened, not when it was imported, so the Overview windows, cost chart buckets, and session
timelines attribute correctly.
* **Near-real-time, not live**: the Data API follows a \~24-hour freshness SLA and retains 30
days - an hourly or daily sync is the honest cadence.
* **No token counts**: Moveworks does not export usage, so these traces contribute nothing to
the LLM cost chart.
* **No fabricated failures**: plugin calls carry served/used flags, not success/failure - the
importer records those flags verbatim and never marks a tool call failed, so Moveworks traffic
cannot generate false tool-failure signals.
* **Deployment-tolerant field mapping**: Data API field names vary between deployments and API
versions; the importer reads each field from a list of known candidates and skips only records
with no parseable timestamp (counted in the report as `skipped_no_time`).
# OpenAI
Source: https://developers.agentx.so/sdk/integrations/openai
Auto-trace raw OpenAI SDK calls with patch_openai_client
For agents built on the higher-level OpenAI Agents SDK instead of the plain client, see [OpenAI Agents SDK](/sdk/integrations/openai-agents).
Install the integration extra:
```bash theme={null}
pip install "agentx-python[openai]"
```
## Usage
Call `patch_openai_client()` once after creating your OpenAI client. All subsequent `client.chat.completions.create()` calls are traced automatically. No changes to individual API calls are needed.
Works with both `openai.OpenAI` and `openai.AsyncOpenAI`.
```python theme={null}
from agentx import AgentX
from agentx.integrations.openai import patch_openai_client
import openai
agentx = AgentX.from_env()
client = openai.OpenAI()
patch_openai_client(
client,
tracer=agentx.tracer,
name="support-agent",
metadata={"env": "production"},
session_id="session-xyz-789",
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
print(response.choices[0].message.content)
agentx.tracer.flush(timeout=10)
```
Async client:
```python theme={null}
patch_openai_client(async_client, tracer=agentx.tracer, name="support-agent")
response = await async_client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
```
## What gets traced
By default, each non-streaming `chat.completions.create()` call produces its own trace.
| Field | Source |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | `messages` kwarg |
| `output` | The choices' message content (or a tool-call description if the reply is a pure tool call) |
| `latencyMs` | Wall-clock time of the API call - measured after the real response comes back, for both sync and async clients |
| `model` | `model` kwarg |
| `inputTokens` / `outputTokens` | `response.usage.prompt_tokens` / `.completion_tokens` |
| `cacheReadTokens` | `response.usage.prompt_tokens_details.cached_tokens` - the cached subset of `inputTokens`, priced at the cache rate. OpenAI has no cache-write concept to report |
| `framework` | Stamped `openai` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `metadata.tools` | The request's `tools=[...]` definitions, captured for the unregistered-tool listing (Tools & MCPs) |
| `error` | Exception message if the API call raises |
Calls made with `stream=True` are passed through untraced. Safely wrapping a
chunk iterator without disrupting the caller's own consumption of it needs
different handling than a single request/response call, so streaming isn't
covered by this integration yet.
The raw OpenAI SDK has no built-in concept of "tool call" or "retrieval"; those only exist as plain Python code around your `chat.completions.create()` calls, so the patch can't see them on its own. Use `tracer.trace_tool_call()` and `tracer.trace_retrieval()` to record them manually so they show up in the trace's performance summary - see the [Anthropic integration's tool-use example](/sdk/integrations/anthropic#tool-use) for the same pattern (identical API, different client).
## Multi-call agentic loops
Like the Anthropic integration, wrap a multi-call tool-use loop in `with tracer.trace(...)` to collapse every `chat.completions.create()` call made inside it into **one** trace instead of one trace per call:
```python theme={null}
with agentx.tracer.trace("support-agent", framework="openai") as span:
span.input = question
# ... call client.chat.completions.create() as many times as needed ...
span.output = final_answer
```
This works because `patch_openai_client()` checks `tracer.current_span` on every call: if a span is active on the current thread, the call is attached to it as an `"LLM Call N"` step; otherwise it sends its own trace as usual.
## `patch_openai_client()` reference
```python theme={null}
patch_openai_client(
client: openai.OpenAI | openai.AsyncOpenAI,
tracer: Tracer,
name: str = "openai-agent",
metadata: dict | None = None,
session_id: str | None = None,
) -> None
```
| Parameter | Description |
| ------------ | ----------------------------------------------------------------- |
| `client` | The `openai.OpenAI()` or `openai.AsyncOpenAI()` instance to patch |
| `tracer` | `agentx.tracer` from your `AgentX` instance |
| `name` | Label shown in the AgentX UI for every trace |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
Calling `patch_openai_client()` on an already-patched client is a no-op; it is safe to call multiple times.
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# OpenAI Agents SDK
Source: https://developers.agentx.so/sdk/integrations/openai-agents
Trace OpenAI agent runs with AgentXTracingProcessor
Install the integration extra:
```bash theme={null}
pip install "agentx-python[openai-agents]" openai-agents
```
## Usage
Register `AgentXTracingProcessor` once at startup. It hooks into the OpenAI Agents SDK's global tracing pipeline, so every subsequent `Runner.run()` call is traced automatically with no per-call changes.
```python theme={null}
from agentx import AgentX
from agentx.integrations.openai_agents import AgentXTracingProcessor
from agents import Agent, Runner, add_trace_processor, function_tool
client = AgentX.from_env() # AGENTX_API_KEY (+ AGENTX_API_BASE_URL for self-host)
add_trace_processor(AgentXTracingProcessor(
tracer=client.tracer,
metadata={"env": "production"},
session_id="session-001",
))
@function_tool
def get_policy(topic: str) -> str:
"""Return the company policy for a given topic."""
db = {
"cancel": "Go to Account → Subscription → Cancel.",
"trial": "14-day free trial, no credit card required.",
"refund": "Full refund within 30 days.",
}
return db.get(topic.lower(), "No policy found.")
agent = Agent(
name="support-agent",
instructions="You are a helpful support agent. Use get_policy to look up policies.",
tools=[get_policy],
)
result = Runner.run_sync(agent, "How do I cancel my subscription?")
print(result.final_output)
client.tracer.flush(timeout=10)
```
## What gets traced
Each top-level agent run (one `Runner.run()` / `Runner.run_sync()` call) produces one trace.
| Field | Source |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input` | The first generation/response span's input (last user message) |
| `output` | The last generation/response span's output (the final reply) |
| `latencyMs` | Wall-clock time from `on_trace_start` to `on_trace_end` |
| `model` | Extracted from the first LLM generation/response span |
| `inputTokens` / `outputTokens` | Summed from each generation/response span's `usage` (`input_tokens` / `output_tokens`) |
| LLM steps | Each `generation`/`response` span becomes a real `"LLM Call N"` child span with its own timing, input/output, model, and tokens - Chat Completions and Responses API both covered |
| Tool steps | Each `function` span becomes a child span named after the tool, with `input`, `output`, and real timing |
| `framework` | Stamped `openai-agents` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `error` | `span.error` from the first failed span in the trace (message + data) |
## `AgentXTracingProcessor` reference
```python theme={null}
AgentXTracingProcessor(
tracer: Tracer,
metadata: dict | None = None,
session_id: str | None = None,
)
```
| Parameter | Description |
| ------------ | ------------------------------------------------- |
| `tracer` | `client.tracer` from your `AgentX` instance |
| `metadata` | Static key-value metadata attached to every trace |
| `session_id` | Links traces from the same conversation thread |
The processor also implements `force_flush()` and `shutdown()`, which both call `tracer.flush()`. These are invoked automatically by the OpenAI Agents SDK at process exit.
Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
In long-running servers it is not required: traces drain automatically in the
background.
# Online Evaluation (Monitor)
Source: https://developers.agentx.so/sdk/monitor
Continuous evaluation of live production traffic - deterministic patterns and LLM-as-judge scoring at ingest, surfaced as triage-ready signals.
**Online evaluation** scores traffic as it happens: every checked trace is evaluated the
moment it arrives, with no dataset and no separate run. Unlike
[offline evaluation](/sdk/evaluations/overview), there is **no ground truth** here - nobody
wrote an expected answer for a live user's question - so verdicts come from LLM-as-judge
scoring and deterministic detectors running against real production behavior, exactly where
your users experience it.
In the dashboard this is the **Monitor** tab. When it finds something, the finding loops back:
a bad trace becomes an offline dataset case in one click, so today's production failure is
tomorrow's regression test.
**Doing it well:**
* **Score the behaviors that hurt in production.** [Online evaluators](/monitor/online-evaluators)
run judge criteria (faithfulness, tool accuracy, goal completion) against sampled live
traces and whole sessions; [patterns](/monitor/patterns) catch the deterministic failures
(errors, empty responses, PII, latency) at zero LLM cost.
* **Tune the sample rate to your volume.** Every judged trace is a real LLM call - start at
`sample_rate=1.0` while traffic is small and you're validating criteria, then dial down as
volume grows. Deterministic patterns stay at 100%: they're free.
* **Let users vote.** [End-user feedback](/monitor/outcomes) (`client.feedback.report`) is the
cheapest evaluation signal you'll ever get - a downvote raises a signal directly, and
[real-world outcomes](/monitor/outcomes) measure how often your judges agreed with reality.
* **Feed findings back into offline tests.** Low-scoring traces (and suspiciously perfect
ones) become [dataset cases](/evaluation/datasets-from-production) - the online/offline loop
is what makes both sides better.
Monitor is AgentX's automatic quality monitoring for production traffic. Every checked trace is evaluated against a set of detectors, and a match becomes a **signal**: a deduped, triage-ready record readable straight from the SDK via `client.monitor.signals`. In the dashboard the machinery splits into two surfaces: **Monitor** is the charts - health strip, healthy-rate and signal-volume trends, and per-scorer live judge score charts with their alert thresholds drawn in (click a card for the scored traces) - while the new **Review** tab is where a human passes verdicts on individual signals: a worst-first queue with the occurrence text and judge rationale as evidence, and Confirm / False positive / Dismiss / Add-to-dataset actions (False positive feeds Judge Calibration and Tune judge). The classic filterable signal table lives under Review > All signals.
A **pattern** is a detection rule with a real id, the same way a dataset or an evaluation config has one. Build a pattern once with `client.monitor.patterns.builder(...).publish()`, then reference it by id, or rely on the built-in checks and any patterns your workspace has defined in the dashboard.
This works the same way for an agent built natively in AgentX and for an external agent traced entirely through the Python SDK.
## Two ways to trigger it
They can be combined; neither requires changes to how you already call `tracer.trace(...)` beyond the flags described below.
### Trace-time: `monitor` and `pattern_ids`
Pass `monitor=True` on `tracer.trace(...)` to check that specific trace immediately, with no dashboard setup at all. `pattern_ids` restricts detection to exactly those patterns; omit it to run the full default sweep (built-in checks plus every active pattern) instead.
`monitor=False` is the opposite: it opts that trace out of **every** ingest-time check - patterns, online evaluators, topic classification. Use it for traces produced inside evaluation runs, where the dataset's own judge already scores each case and a second judging pass would only double the bill. Leaving `monitor` unset keeps the engine's normal behavior.
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
pattern = client.monitor.patterns.builder(
name="Promises a refund",
detector_kind="semantic",
semantic_prompt="The response promises a refund.",
severity="high",
).publish()
with client.tracer.trace(
"support-agent", monitor=True, pattern_ids=[pattern.id]
) as span:
span.output = call_llm(query)
```
`pattern_ids` fully defines what's checked when provided: only those named
patterns run, the built-in checks are skipped. This mirrors how
`scorer_id` fully defines an evaluation's grading config rather
than layering on top of a default. Omit `pattern_ids` (keep just
`monitor=True`) to run the full default sweep instead.
### Dashboard toggle: automatic, every trace from an agent
Enable monitoring once per agent in the dashboard, and every subsequent trace from that agent is checked automatically, with no `monitor=True` needed on any individual call.
1. **Send at least one trace.** The first call to `tracer.trace(...)` for a given agent name auto-creates a reference agent in your workspace. See [Tracing](/sdk/tracing) if you haven't wired this up yet.
2. **Open Governance > Agents.** Your SDK-traced agent appears in the agent list with an **External** badge, alongside any native agents.
3. **That's it - monitoring is on.** Every ingested trace is checked by the scorers you enable. There is no global sample rate: sampling lives on the scorers that spend LLM budget (each judge scorer's own rate, the Topics classification rate), so free deterministic checks always see all traffic.
## What gets checked
Three separate streams classify a monitored trace:
**Operational outcomes** - facts the trace itself recorded, always on, never configurable, and
never raising triage signals (they feed the KPI failure metrics and Top failing instead):
| Operational outcome | Recorded when |
| ------------------- | ---------------------------------------------------------------------------------------------------------- |
| Tool/action failure | A recorded tool call reports `success: false` |
| Trace error | `error` was set on the trace (e.g. via `span.set_error(...)`, or an exception inside a decorated function) |
| Empty response | The traced output is empty |
Latency is a distribution metric (Overview's p95 card, straight from the traces), not a failure
classification.
**Scorers** - judgments you opt in to from the [Scorers page](/monitor/scorers), all off by default. Six built-in
template scorers ship with the engine, all zero-LLM-cost; your own template patterns, LLM judge
scorers, and custom endpoint scorers layer on top:
| Built-in template scorer | Flags when the response contains | Severity |
| ------------------------ | ------------------------------------------------------------------------------------------------------------ | -------- |
| Secrets in response | A leaked credential: API keys (OpenAI/AWS/GitHub/Slack), bearer tokens, JWTs, private-key blocks | critical |
| PII in response | Personal data: emails, phone numbers, SSNs, payment cards | high |
| Prompt injection echo | A known jailbreak phrasing repeated in the output ("ignore previous instructions", system-prompt disclosure) | high |
| Profanity in response | Profanity from a small unambiguous wordlist | medium |
| Refusal / non-answer | A deflection ("I can't help with that") - sometimes correct behavior, surfaced for triage | low |
| Malformed JSON response | Output that isn't parseable JSON - enable only for agents whose contract is JSON | medium |
**Human feedback** - your users' votes via `client.feedback.report(trace_id, "down")`, raising a
*Negative user feedback* signal directly (see [User Feedback](/monitor/user-feedback)).
Metric semantics: operational outcomes and template/pattern hits classify the RUN (they move
`failureRate` and the run-outcome breakdown); judge, code, and external scorer verdicts are
evaluator events - they raise signals and keep per-scorer histories without reclassifying the
run. See [what moves which metric](/monitor/scorers#what-moves-which-metric).
## Scorer administration as code
Everything the Scorers page does is scriptable via `client.monitor.scorers` - enable the
shipped templates, deploy code/external scorers, and read their check history:
```python theme={null}
# Templates are opt-in; enable() preserves what's already on and returns the resulting list.
client.monitor.scorers.enable(["pii-in-response", "secrets-in-response"])
client.monitor.scorers.templates() # keys, rules, enabled state
scorer = client.monitor.scorers.create_code(
"Apology overload",
"async def handler(input, output, expected, metadata, trace):\n"
" return 0.0 if str(output).lower().count('sorry') >= 2 else 1.0\n",
language="python", sample_rate=0.5, alert_below=0.5,
)
client.monitor.scorers.dry_run(kind="code", language="python", script="...") # nothing persisted
client.monitor.scorers.events(scorer["_id"], window="24h") # per-check history
client.monitor.scorers.create_external("Policy endpoint", "https://scorer.internal/score")
```
LLM judges have their own unified surface, `client.monitor.judge_scorers`: one scorer = one
judge rubric + an offline profile (dataset-run grading; the scorer's `id` is exactly the
`scorer_id` a run takes - `evaluation_settings_id` remains a working alias) + an optional online profile (live-traffic scoring).
Strictly one online profile per scorer.
```python theme={null}
scorer = client.monitor.judge_scorers.create(
"Support quality",
judge={"acceptanceCriteria": "Concrete, correct, cites the policy."},
offline={"numberOfRequests": 2, "jaccardSimilarity": {"enabled": True}},
online={"enabled": True, "sampleRate": 0.2, "alertThreshold": 6},
)
client.evaluations.run(dataset_id, subject, scorer_id=scorer.id) # same rubric offline
# Or the builder, the unified successor of evaluations.settings.builder - rubric,
# offline profile, and live profile in one call:
scorer = client.monitor.judge_scorers.builder(
"Support quality",
acceptance_criteria="Concrete and correct.",
tool_context="detailed",
number_of_requests=2,
live=True, sample_rate=0.25,
).publish()
client.monitor.judge_scorers.update(scorer.id, online={"enabled": False}) # sparse: rubric untouched
client.monitor.judge_scorers.update(scorer.id, online=None) # detach the online profile
client.monitor.judge_scorers.calibration(scorer.id) # tuning/ratings/events resolve the profile id
```
The older per-profile surfaces (`client.evaluations.settings`, `client.monitor.online_evaluators`)
keep working unchanged as views onto the same entity - each emits a default-hidden
`DeprecationWarning` on first use pointing here.
Projects and trace read-back are first-class too: `client.projects.create(name)` returns an
isolated project with its own `apiKey` (the per-run isolation pattern integration suites use),
and `client.traces.get(trace_id)` / `client.traces.list(cursor=...)` read what the tracer wrote.
Pattern scorers you author are rules matched against the response text or the trace: keyword/`contains`, `regex`, or an LLM-judged semantic rubric. Build them via the dashboard (Scorers → **New scorer** → **Pattern scorer** - see [Patterns](/monitor/patterns) for the condition builder, AND/OR/NOR combination, and match targets) or with `client.monitor.patterns.builder(...)`, shown below.
The "Tool/action failure" check reads the `success: false` a recorded tool
call carries. The easiest way to produce that is
`tracer.trace_tool_call(...)`, which captures an escaping exception as a
failed call automatically - see [Tracing](/sdk/tracing#context-manager).
A trace that matches nothing becomes a healthy "info" tally instead, which powers the agent's health-rate percentage.
## `client.monitor.patterns`
```python theme={null}
pattern = client.monitor.patterns.builder(
name="Promises a refund",
detector_kind="semantic",
semantic_prompt="The response promises a refund.",
).publish()
print(pattern.id)
client.monitor.patterns.get(pattern.id) # fetch one
client.monitor.patterns.list() # list every pattern in the workspace
```
### `builder()` parameters
| Parameter | Type | Default | Description |
| --------------------------------- | ------------------- | -------------- | -------------------------------------------------------------------------------- |
| `name` | `str` | required | Pattern display name |
| `description` | `str` | none | Human-readable description |
| `detector_kind` | `str` | `"contains"` | `"contains"`, `"regex"`, or `"semantic"`, selects which field below is used |
| `match_target` | `list[str]` | `["response"]` | Where to look: `"response"`, `"userMessage"`, `"trace"` |
| `match_mode` | `str` | `"any"` | For `detector_kind="contains"`: `"any"` or `"all"` of `include_terms` must match |
| `include_terms` / `exclude_terms` | `list[str]` | `[]` | Phrases to require / exclude, for `detector_kind="contains"` |
| `regex` | `str` | none | Regular expression body, for `detector_kind="regex"` |
| `semantic_prompt` | `str` | none | Rubric an LLM judges the response against, for `detector_kind="semantic"` |
| `severity` | `str` | `"medium"` | `"low"`, `"medium"`, `"high"`, or `"critical"` |
| `polarity` | `str` | `"failure"` | `"failure"` raises a signal to triage; `"proper"` logs a healthy tally instead |
| `enabled` | `bool` | `True` | Whether the pattern is checked at all |
| `sample_rate` | `float` | `1.0` | Fraction of matching traces to actually check, `0.0` to `1.0` |
| `scope_mode` / `agent_ids` | `str` / `list[str]` | `"all"` / `[]` | Restrict this pattern to specific agents instead of the whole workspace |
`publish()` returns a pattern with `.id`, which you pass in `pattern_ids` at trace time.
On the hosted platform, creating a pattern requires a Business or Enterprise
plan (the same entitlement gate as the dashboard's Patterns UI). Self-host has
no plan gates.
## Where signals show up
Matches are deduped by pattern and (usually) by agent, so a recurring issue accumulates occurrences on one signal instead of creating a new row every time. Each signal links back to the trace and tool calls that produced it, so a reviewer can open the exact exchange from the dashboard (Governance > Observe), or read it straight from the SDK.
## `client.monitor.signals`
Read-only: a signal is the system's output from checking traces against patterns, not something you create directly.
```python theme={null}
signals = client.monitor.signals.list(severity="high", limit=20)
for s in signals:
print(s.id, s.severity, s.summary, s.occurrence_count)
signal = client.monitor.signals.get(signals[0].id)
print(signal.recommended_actions)
```
### `list()` parameters
| Parameter | Type | Default | Description |
| ---------- | ----- | ------------------------------ | ------------------------------------------------------------------------------------------------ |
| `polarity` | `str` | server default (failures only) | `"failure"`, `"proper"` (healthy tally), or `"all"` for both |
| `status` | `str` | none | e.g. `"open"`, filter to one status |
| `severity` | `str` | none | `"low"`, `"medium"`, `"high"`, or `"critical"` |
| `agent_id` | `str` | none | Restrict to one agent (matches either the signal's representative agent or its occurrence trail) |
| `limit` | `int` | `50` | Capped at 100 server-side |
`list()`/`get()` both return a signal with `.id`, `.type`, `.severity`, `.polarity`, `.status`, `.summary`, `.pattern_key`, `.occurrence_count`, `.occurrences`, `.recommended_actions`, `.root_cause`, and more, matching the fields shown in the dashboard's triage queue.
## Reading production metrics
The numbers behind the dashboard's Overview and Monitor charts are readable as plain dicts,
for wiring into your own dashboards and alerts:
```python theme={null}
client.monitor.kpis(window="7d") # Overview KPI strip: totalRuns, healthRate,
# failureRate, downvoteRate, toolFailureRate,
# p95LatencyMs, deltas vs the prior window
client.monitor.metrics(window="1d") # Monitor metrics grid: bucketed spans by kind,
# latency percentiles, tokens/cost, tool executions
# and failures, platform attribution
client.monitor.calibration(window="7d") # Judge Calibration: comparedCount, agreementRate,
# falsePositiveRate, falseNegativeRate
```
`kpis()` and `calibration()` take a `window` of `"24h"`, `"7d"` (default), or `"30d"`.
### `metrics()` filters
`metrics()` accepts a `window` from `"1h"` to `"90d"` (default `"1d"`) plus optional filters
that scope every number the way the dashboard's filter chips do:
```python theme={null}
client.monitor.metrics(window="7d", agent="support-agent", status="error")
client.monitor.metrics(window="1d", framework="langchain") # one platform's traffic
```
| Filter | Description |
| ----------- | ------------------------------------------------------------------------------------------------------------- |
| `agent` | Only traffic from this agent |
| `model` | Only spans that ran on this model |
| `tool` | Only executions of this tool |
| `framework` | The platform label traces carry (see [Platform detection](/sdk/tracing)); `"other"` selects unlabeled traffic |
| `status` | Run status, e.g. `"error"` |
The response includes platform attribution - `frameworks` window totals plus per-bucket
`byFramework` - which is the data behind the Platforms chart.
Two related project-wide switches: `client.monitor.topics(window)` reads the
[Topics](/monitor/topics) view (LLM-classified themes of sampled traffic - empty until enabled),
and `client.monitor.set_topics(True, sample_rate=0.2)` turns classification on (each classified
trace costs one judge call, hence the sample rate).
## Human review and automation rules
The review loop and the routing rules the dashboard offers are scriptable too:
* **`client.monitor.review_queue`** - `list()`, `queue(trace_id)`, `label(item_id, "good"|"bad",
corrected_score=...)`, `dismiss(item_id)`. Labels feed judge calibration and become
judge-tuning evidence. See [Review queue](/monitor/review-queue).
* **`client.monitor.rules`** - `create(name, action, filter=..., sample_rate=...)` where
`action` is `"review"` (sample matches into the review queue), `"dataset"` (append matches
as dataset cases), or `"webhook"` (POST the matching trace to your URL); plus `list()`,
`update()`, `delete()`. See [Rules](/monitor/rules).
## `client.monitor.profile`
Get/update one agent's monitoring profile: enable/disable, detection category opt-outs, and approval policy. The `coverage_mode`/`sample_rate` pair still round-trips for wire compatibility but gates nothing - detection runs on all ingested traffic, and sampling lives on each scorer's own rate. Legacy `threshold_overrides` fields also still round-trip; latency is a KPI metric now (Overview's p95 card, straight from the traces) rather than a detection threshold.
```python theme={null}
profile = client.monitor.profile.get("agent_123")
print(profile.coverage_mode if profile else "never configured, on defaults")
client.monitor.profile.update("agent_123", coverage_mode="all", sample_rate=0.5)
```
`get()` returns `None` when the agent has never been configured (still on platform defaults). `update()` upserts and only changes the fields you pass:
| Parameter | Type | Description |
| ------------------------------------------------------ | ---------------- | ----------------------------------------------------------- |
| `enabled` | `bool` | Turn Monitor on/off for this agent |
| `failure_detection_enabled` / `info_detection_enabled` | `bool` | Opt a whole detection category out |
| `coverage_mode` | `str` | Legacy, gates nothing - detection always sees every trace |
| `sample_rate` | `float` | Legacy, gates nothing - use each scorer's own `sample_rate` |
| `channels` | `list[str]` | Notification channels |
| `dataset_id` | `str` | Evaluation dataset this agent's signals feed into |
| `threshold_overrides` | `dict` | Per-check threshold overrides, e.g. `{"latencyMs": 15000}` |
| `retention_days` | `int` | How long monitored traces are kept |
| `approval_policy` | `dict[str, str]` | Per-action approval mode for autotune actions |
## Self-host extras
A [self-hosted](/self-host/overview) instance adds three Monitor surfaces the hosted platform doesn't have yet, both reachable from this same SDK:
* **Live LLM judge scoring** - a real LLM judge scoring a sample of live traffic continuously, per trace or per whole session (`scope="session"`, judged automatically once the conversation goes idle). Configure it as the online profile of a [judge scorer](#scorer-administration-as-code) (`client.monitor.judge_scorers`, or the legacy `client.monitor.online_evaluators` view). See [Online evaluators](/monitor/online-evaluators).
* **Scorer groups** (`client.monitor.scorer_groups`) - scorers of any kind composed into one weighted 0-10 score with must-pass gates, usable as a dataset run's grader and against live traffic, per trace or per whole session. Session helpers live on `client.monitor.sessions` (`scores()`, `run_sweep()`, `spans()`, `coherence_check()`). See [Scorer groups](/monitor/scorer-groups).
* **Outcome reports and user feedback** (`client.outcomes.report(...)` / `client.feedback.report(...)`) - record what actually happened after the fact (a reopened ticket, an end user's thumbs-down) against a trace; negative feedback raises a signal directly, and both streams feed the dashboard's Judge Calibration card. See [Outcomes & Judge Calibration](/monitor/outcomes).
Send the traces this feature monitors
The condition builder, match targets, and where signals go
# Python SDK Overview
Source: https://developers.agentx.so/sdk/overview
One client for tracing, monitoring, evaluations, prompts, and CI gates
## Installation
```bash theme={null}
pip install agentx-python
```
Requires Python 3.9+. Framework extras pull in only what each integration needs, e.g.
`pip install "agentx-python[langchain]"` - see [Framework Integrations](/sdk/integrations/langchain).
## Initialization
```python theme={null}
from agentx import AgentX
# From environment: AGENTX_API_KEY (+ AGENTX_API_BASE_URL for self-host)
client = AgentX.from_env()
# Explicit
client = AgentX(
api_key="agtx_local_...",
base_url="http://localhost:4700/api/v1", # your self-host engine
)
```
The key comes from the engine's startup log or **Settings → API access** - see
[Authentication](/authentication).
Construction is lazy (no network call), and tracing is fire-and-forget - a wrong URL or key
surfaces as a one-time log warning, not an exception. To fail fast instead, verify once at
startup:
```python theme={null}
client.ping()
# AgentXConnectionError: Cannot reach AgentX at http://fake-url.com/api/v1 (...)
# AgentXAuthError: AgentX at ... rejected the API key (HTTP 401). Check api_key / AGENTX_API_KEY ...
```
## The client at a glance
| Module | What it does | Guide |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `client.tracer` | Record traces: decorator, context manager, sessions, tool calls, span trees | [Tracing](/sdk/tracing) |
| `agentx.integrations.*` | One-line capture for LangChain/LangGraph, OpenAI Agents, CrewAI, AutoGen, ADK, LlamaIndex, Databricks/MLflow, Moveworks, and raw OpenAI/Anthropic clients | [Integrations](/sdk/integrations/langchain) |
| `client.evaluations` | Datasets, runs (`run(dataset_id, subject, scorer_id=...)` + `.execute(fn).finalize()`), prompts, tool schemas | [Evaluations](/sdk/evaluations/overview) |
| `client.evaluations.prompts` | The prompt registry: pull versioned prompts at runtime | [Prompt Management](/improve/prompt-management) |
| `client.monitor` | Patterns, LLM Judge Scorers, the review queue, automation rules, KPIs/metrics, and per-agent profiles from code | [Monitor](/sdk/monitor) |
| `client.outcomes` / `client.feedback` | Report real-world outcomes and end-user votes for judge calibration | [Outcomes](/monitor/outcomes) |
| `client.projects` / `client.traces` | Project CRUD and the trace read side (by id, paginated lists) - self-host | [Monitor](/sdk/monitor) |
| `client.export` | Bulk NDJSON export for backup/migration: manifest, streaming, directory dumps | [Backup & export](/self-host/backup) |
| `run.gate()` / `run_eval()` | CI pass/fail gates with rating floors and no-regression checks | [CI/CD](/sdk/ci-cd) |
| CLI: `agentx-moveworks`, `agentx-databricks` | Pull importers replaying external platforms' activity as traces | [Moveworks](/sdk/integrations/moveworks) · [Databricks](/sdk/integrations/databricks) |
## Guides
Record agent runs - spans, sessions, tool calls
Score any agent against a dataset in one script
Continuous detection on production traffic
Gate releases on measured quality
# Python SDK Reference
Source: https://developers.agentx.so/sdk/python-reference
Complete API reference for agentx-python
## `AgentX`
The top-level client. Create one per process.
```python theme={null}
from agentx import AgentX
client = AgentX(api_key="a1b2c3d4e5f60718293a4b5c6d7e8f9012345678", base_url="http://localhost:4700/api/v1")
# or from environment:
client = AgentX.from_env() # reads AGENTX_API_KEY, plus the first of
# AGENTX_API_BASE_URL / AGENTX_SELFHOST_BASE_URL / BASE_URL
```
### Constructor
| Parameter | Type | Default | Description |
| -------------- | ----- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| `api_key` | `str` | `AGENTX_API_KEY` env var | Workspace API key |
| `base_url` | `str` | `AGENTX_API_BASE_URL` or SDK default | Override API base URL (self-hosted). Include the `/api/v1` suffix, e.g. `http://localhost:4700/api/v1` |
| `workspace_id` | `str` | `AGENTX_WORKSPACE_ID` env var | Explicit workspace ID, required when the API key belongs to a different user than the workspace being targeted |
### Class methods
| Method | Returns | Description |
| ------------------- | -------- | ---------------------------------------------------- |
| `AgentX.from_env()` | `AgentX` | Construct from `AGENTX_API_KEY` environment variable |
### Attributes
| Attribute | Type | Description |
| ------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `client.tracer` | `Tracer` | Tracer and CI/CD evaluation |
| `client.evaluations` | `EvaluationsRunner` | Dataset-driven evaluation runner, prompts, tool schemas |
| `client.monitor` | `MonitorClient` | [Monitor](/sdk/monitor): patterns, signals, judge scorers, review queue, rules, KPIs/metrics, profiles |
| `client.outcomes` / `client.feedback` | `OutcomesClient` / `FeedbackClient` | Report real-world outcomes and end-user votes against traces (self-host) - see [Outcomes](/monitor/outcomes) |
| `client.projects` | `ProjectsClient` | Project CRUD with per-project API keys (self-host) |
| `client.traces` | `TracesClient` | Read side of tracing: trace by id, paginated listing (self-host) |
| `client.export` | `ExportClient` | Bulk NDJSON export for backup/migration (self-host) - see [Backup & export](/self-host/backup) |
***
## `Tracer`
Accessed via `client.tracer`. Handles both tracing and CI/CD evaluation.
### `trace()`
```python theme={null}
tracer.trace(
name: str,
*,
input: Any = None,
metadata: dict | None = None,
framework: str | None = None,
model: str | None = None,
session_id: str | None = None,
sync: bool = False,
monitor: bool | None = None,
pattern_ids: list[str] | None = None,
agent_id: str | None = None,
span_kind: str | None = None,
) -> _TraceSpan
```
Returns a `_TraceSpan` that works as a **decorator** or **context manager**.
| Parameter | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Agent or operation label shown in the UI |
| `input` | Initial input (context manager only; overrides captured function args) |
| `metadata` | Arbitrary key-value metadata (not indexed, max 16 KB) |
| `framework` | `"langchain"`, `"crewai"`, `"openai-agents"`, `"anthropic"`, or custom |
| `model` | LLM model name, e.g. `"gpt-4o"` |
| `session_id` | Groups traces from the same session or conversation thread |
| `sync` | Context manager only. Sends synchronously (blocking) instead of the default fire-and-forget queue, so `span.trace_id` is populated once the block exits. See [Linking a trace to an evaluation result](/sdk/tracing#linking-a-trace-to-an-evaluation-result). |
| `monitor` | `True`: check this trace against Monitor patterns immediately. `False`: opt out of every ingest-time check (patterns, online evaluators, topics) - for traces made inside evaluation runs. `None` (default): normal engine behavior. |
| `pattern_ids` | With `monitor=True`, restrict detection to exactly these pattern ids (built-ins skipped). |
| `agent_id` | Pin this trace to a known agent id instead of resolving the agent by `name`. |
| `span_kind` | Span classification shown in the timeline, e.g. `"llm"`, `"tool"`, `"retrieval"`. |
### `flush()`
```python theme={null}
tracer.flush(timeout: float = 5.0) -> bool
```
Blocks until all queued traces have been delivered, or until `timeout` seconds elapse. Returns `True` when everything drained, `False` on deadline (a warning is logged and undelivered traces keep sending in the background). Call before process exit in scripts or one-shot jobs.
### `use_span()`
```python theme={null}
tracer.use_span(span: _TraceSpan) -> ContextManager[_TraceSpan]
```
Makes `span` (created on another thread) the active span for the duration of the block, on the *calling* thread.
The active-span stack is thread-local, so a span opened with `tracer.trace(...)` on the main thread isn't automatically visible inside a `ThreadPoolExecutor` worker (or any other thread). Wrap the worker's body in `use_span()` to attach its LLM and tool calls to the parent span instead of starting an independent trace.
```python theme={null}
with tracer.trace("orchestrator") as span:
def worker():
with tracer.use_span(span):
chain.invoke(query, config={"callbacks": [handler]})
with ThreadPoolExecutor(max_workers=2) as ex:
ex.submit(worker).result()
```
Safe to call concurrently from multiple threads for the same span: each thread pushes and pops on its own stack.
### `run_eval()`
```python theme={null}
tracer.run_eval(
dataset_id: str,
agent_fn: Callable[[str], str],
*,
agent_name: str | None = None,
pass_rate_threshold: float | None = None,
git_context: dict | None = None,
concurrency: int = 1,
fail_on_gate: bool = False,
timeout_per_question: float | None = None,
) -> CIRunResult
```
High-level CI/CD evaluation: creates a run, calls `agent_fn` for every test case, submits results, finalizes, and returns the gate decision.
Raises `CIGateFailure` (containing the `CIRunResult`) when `fail_on_gate=True` and the gate is `"fail"`.
### `create_ci_run()`
```python theme={null}
tracer.create_ci_run(
dataset_id: str,
*,
agent_name: str | None = None,
pass_rate_threshold: float | None = None,
git_context: dict | None = None,
workspace_id: str | None = None,
) -> CIRun
```
### `submit_result()`
```python theme={null}
tracer.submit_result(
run_id: str,
question_index: int,
output: Any,
*,
input: Any = None,
latency_ms: int | None = None,
) -> CIQuestionScore
```
### `finalize_ci_run()`
```python theme={null}
tracer.finalize_ci_run(run_id: str) -> CIRunResult
```
### `get_ci_run()`
```python theme={null}
tracer.get_ci_run(run_id: str) -> CIRunStatus
```
### `evaluate_trace()`
```python theme={null}
tracer.evaluate_trace(
trace_id: str,
dataset_id: str,
*,
question_index: int | None = None,
) -> dict
```
Score a previously-recorded trace against a dataset. The agent is not re-run. Returns `{ run_id, trace_id, rating, justification, status }`. `trace_id` comes from a prior `tracer.trace(..., sync=True)` call's `span.trace_id`.
***
## `_TraceSpan`
Returned by `tracer.trace()`. Can be used as a decorator or context manager.
### Attributes
| Attribute | Type | Description |
| ----------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `span.input` | `Any` | Agent input. Assign in context manager to override captured args. |
| `span.output` | `Any` | Agent output. Must be set manually in context manager mode. |
| `span.tool_calls` | `list` | Accumulated tool calls. Append with `add_tool_call()`. |
| `span.trace_id` | `str \| None` | The ingested trace's id, readable once the block exits. Only populated when opened with `sync=True`; `None` otherwise, since the default mode queues the send and never learns the id. |
### Methods
#### `add_tool_call()`
```python theme={null}
span.add_tool_call(
name: str,
*,
input: Any = None,
output: Any = None,
latency_ms: int | None = None,
) -> None
```
Records a tool call made during this span. Safe to call multiple times.
#### `set_error()`
```python theme={null}
span.set_error(message: str) -> None
```
Marks this span as failed. Takes precedence over any exception automatically caught by `__exit__`.
### Usage patterns
```python Decorator theme={null}
@tracer.trace("my-agent", framework="langchain", model="gpt-4o")
def run(query: str) -> str:
return chain.invoke(query)
# input = captured from function args
# output = return value
```
```python Context manager theme={null}
with tracer.trace("my-agent") as span:
span.input = {"query": query, "user_id": user_id}
kb = search_kb(query)
span.add_tool_call("search_kb", input=query, output=kb, latency_ms=180)
span.output = llm.invoke(f"Context: {kb}\n\n{query}")
```
```python Get trace_id back theme={null}
with tracer.trace("my-agent", sync=True) as span:
span.output = call_agent(query)
print(span.trace_id) # None unless sync=True
```
```python Async decorator theme={null}
@tracer.trace("async-agent", framework="openai-agents")
async def run(query: str) -> str:
result = await runner.run(agent, query)
return result.final_output
```
```python Error handling theme={null}
with tracer.trace("my-agent") as span:
try:
span.output = call_agent(query)
except Exception as e:
span.set_error(str(e))
raise
```
***
## CI/CD Dataclasses
### `CIRun`
Returned by `create_ci_run()`.
```python theme={null}
@dataclass
class CIRun:
run_id: str
dataset_id: str
total_questions: int
test_cases: list[CITestCase]
expires_at: str # ISO 8601, run expires if not finalized within 2 hours
```
### `CITestCase`
```python theme={null}
@dataclass
class CITestCase:
index: int
query: str | None # None when ci.exposeTestInputs is False
```
### `CIQuestionScore`
Returned by `submit_result()`.
```python theme={null}
@dataclass
class CIQuestionScore:
question_index: int
rating: int # 0-10
justification: str
passed: bool
gate_fired: bool # True when failFast fired, stop submitting
input: Any
output: Any
```
### `CIRunResult`
Returned by `finalize_ci_run()` and `run_eval()`.
```python theme={null}
@dataclass
class CIRunResult:
run_id: str
gate: Literal["pass", "fail"]
pass_rate: float # 0.0 - 1.0
total_questions: int
passed_questions: int
scores: list[CIQuestionScore]
violations: list[ThresholdViolation]
git_context: dict | None
finalized_at: str | None
```
### `CIRunStatus`
Returned by `get_ci_run()`.
```python theme={null}
@dataclass
class CIRunStatus:
run_id: str
status: Literal["in_progress", "completed", "failed"]
gate: Literal["pass", "fail"] | None # None while in_progress
results_submitted: int
total_questions: int
created_at: str
expires_at: str
finalized_at: str | None
git_context: dict | None
```
### `ThresholdViolation`
```python theme={null}
@dataclass
class ThresholdViolation:
question_index: int
metric: str # e.g. "rating"
threshold: float
actual: float
question_text: str
```
***
## Exceptions
All exceptions inherit from `AgentXError`.
```python theme={null}
from agentx import (
AgentXError,
AgentXAuthError,
AgentXAPIError,
DatasetNotFound,
CINotEnabled,
CIRunExpired,
CIGateFailure,
)
```
| Exception | When raised |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `AgentXError` | Base class for all SDK errors |
| `AgentXAuthError` | Invalid or missing API key |
| `AgentXConnectionError` | The API (or self-host engine) could not be reached at the configured `base_url`, e.g. from `client.ping()`. Import from `agentx.exceptions`. |
| `AgentXAPIError` | Unexpected API error. Has `.status_code: int \| None` attribute. |
| `DatasetNotFound` | Dataset ID does not exist or is not accessible |
| `CINotEnabled` | Dataset exists but `ci.enabled` is `False` |
| `CIRunExpired` | CI run was not finalized within 2 hours |
| `CIGateFailure` | Gate is `"fail"` and `fail_on_gate=True`. Has `.result: CIRunResult` attribute. |
### `CIGateFailure`
```python theme={null}
try:
result = client.tracer.run_eval(
dataset_id=dataset_id,
agent_fn=my_agent,
fail_on_gate=True,
)
except CIGateFailure as e:
print(f"Pass rate: {e.result.pass_rate:.0%}")
for v in e.result.violations:
print(f" Q{v.question_index}: {v.metric} = {v.actual:.2f} (threshold {v.threshold:.2f})")
sys.exit(1)
```
***
## Environment variables
| Variable | Description |
| --------------------- | ---------------------------------------------------------------- |
| `AGENTX_API_KEY` | Workspace API key (required) |
| `AGENTX_API_BASE_URL` | Override API base URL for self-hosted deployments |
| `AGENTX_WORKSPACE_ID` | Explicit workspace ID, injected into every request automatically |
# Tracing
Source: https://developers.agentx.so/sdk/tracing
Record agent runs from any Python framework
The `Tracer` captures your agent's inputs, outputs, latency, and tool calls, with a single decorator or context manager. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets.
Running [self-hosted](/self-host/overview)? If your app is already instrumented with OpenTelemetry, you don't need this SDK to get traces in at all - point your OTel exporter straight at the engine. See [Connect via OpenTelemetry](/trace/opentelemetry).
## Decorator
The simplest usage. AgentX captures function arguments as `input`, the return value as `output`, and wall-clock time as `latencyMs`.
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
tracer = client.tracer
@tracer.trace("customer-support-agent", framework="langchain", model="gpt-4o")
def handle(query: str) -> str:
return chain.invoke(query)
handle("How do I reset my password?")
```
Works with async functions too:
```python theme={null}
@tracer.trace("async-agent", framework="openai-agents")
async def handle_async(query: str) -> str:
result = await runner.run(agent, query)
return result.final_output
```
## Context manager
Use when you need to set input/output manually or record tool calls mid-span.
```python theme={null}
with tracer.trace("rag-agent", framework="langchain") as span:
span.input = {"query": query, "user_id": user_id}
kb_result = search_knowledge_base(query)
span.add_tool_call("search_knowledge_base", input=query, output=kb_result, latency_ms=190)
answer = llm.invoke(f"Context: {kb_result}\n\nQuery: {query}")
span.output = answer
```
To time a tool call and capture its failures automatically, wrap the execution itself with `tracer.trace_tool_call()` instead of reporting it after the fact:
```python theme={null}
with tracer.trace_tool_call("search_knowledge_base", input=query) as t:
t.output = search_knowledge_base(query)
```
An exception escaping the block records the call as failed (`success=False` plus the error text - what Monitor's built-in "Tool failure" check reads) and then propagates unchanged. To set the outcome yourself, use `tracer.record_tool_call(name, input=..., output=..., success=False, error="...")`.
### `_TraceSpan` attributes and methods
| Name | Description |
| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `span.input = value` | Override the captured input |
| `span.output = value` | Set the output (required in context manager mode) |
| `span.add_tool_call(name, *, input, output, latency_ms)` | Record a tool call made during this span |
| `span.set_error(message)` | Mark the span as failed with a custom error message |
| `span.trace_id` | The ingested trace's id, once the block has exited. Only populated when opened with `sync=True` (see below) |
On a root span, `sync=True` covers the **whole tree**: child spans recorded inside the block
(tool calls, LLM calls) are drained before the root is sent, so a read immediately after the
block sees every span. Child-only spans keep their async fire-and-forget behavior;
`client.tracer.flush()` remains available for manual control.
### Decorator vs. context manager
Both forms capture a usable trace, including at least one Execution Timeline step synthesized from the wrapped call's input/output if nothing more granular was recorded. They differ in what you get back:
| | Decorator | Context manager |
| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| Sends the trace | Fire-and-forget by default, or blocking with `sync=True` | Fire-and-forget by default, or blocking with `sync=True` |
| Can retrieve `trace_id` | **No.** The wrapped function returns its own value, not a span - even with `sync=True` there is no span object to read the id from | **Yes**, with `sync=True` |
| Manual enrichment (`add_tool_call`, `record_retrieval`, multiple steps) | No, one function call produces one synthesized step | Yes, call these anywhere inside the block |
| Best for | Fire-and-forget background tracing where nothing downstream needs the id in the same call | Anything that needs the id afterward, e.g. [linking a trace to an evaluation result](#linking-a-trace-to-an-evaluation-result), or `evaluate_trace()` |
If you don't need the id back, the decorator is simplest. If you do, use the context manager with `sync=True`. It adds one blocking network round-trip (typically well under a second) in exchange for the id being ready the moment the `with` block exits.
## Parallel work across threads
The active span is tracked per-thread. If you fan work out to a `ThreadPoolExecutor` (or any other thread) from inside a `tracer.trace(...)` block, worker threads don't automatically see the span opened on the calling thread, so each LLM or tool call inside them would start its own independent trace instead of landing as a step on the parent span.
Wrap each worker's body in `tracer.use_span(span)` to attach it to the parent span for the duration of that block:
```python theme={null}
with client.tracer.trace("billing-dispute-orchestrator") as orch:
orch.input = user_message
# Policy and risk specialists run in parallel, each on its own thread.
# `orch` was opened on the main thread, so each worker must explicitly
# attach to it via `use_span`, otherwise these steps become their own
# separate traces instead of landing on the orchestrator trace.
def run_policy_specialist():
with client.tracer.use_span(orch):
return chain.invoke(policy_query, config={"callbacks": [handler]})
def run_risk_specialist():
with client.tracer.use_span(orch):
return chain.invoke(risk_query, config={"callbacks": [handler]})
with ThreadPoolExecutor(max_workers=2) as ex:
fut_policy = ex.submit(run_policy_specialist)
fut_risk = ex.submit(run_risk_specialist)
policy_result = fut_policy.result()
risk_result = fut_risk.result()
orch.output = combine(policy_result, risk_result)
```
`use_span` is safe to call concurrently from multiple threads for the same span. Each thread pushes and pops on its own stack, so parallel workers don't interfere with each other.
## `tracer.trace()` parameters
Every parameter works in both decorator and context-manager mode - the decorator forwards
`sync`, `monitor`, `pattern_ids`, `agent_id`, and `span_kind` to each call's span the same way
the context manager does.
| Parameter | Type | Description |
| ------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | Agent or operation label shown in the UI. Traces resolve to one stable agent per distinct name |
| `input` | `Any` | Initial input for the span. In decorator mode the function's arguments are captured automatically; in context-manager mode you can also assign `span.input` inside the block |
| `framework` | `str` | The platform label: `"langchain"`, `"crewai"`, `"openai-agents"`, or any custom string (`"my-inhouse-runner"`). Left unset, [integrations](/sdk/integrations/langchain) stamp their own literal, and otherwise the SDK [auto-detects](/trace/platform-detection) the single orchestration framework imported in the process - ambiguous stays unlabeled |
| `model` | `str` | LLM model used, e.g. `"gpt-4o"`, `"claude-sonnet-4-6"` |
| `session_id` | `str` | Groups traces from the same user session or thread |
| `metadata` | `dict` | Arbitrary key-value metadata (not indexed, max 16 KB) |
| `agent_id` | `str` | Pins this trace to an exact agent id (one you already know, e.g. from the dashboard) instead of resolving by `name`. Omit it and the trace resolves from `name` alone |
| `sync` | `bool` | `False` (default) queues the trace on a background thread. Never blocks, but `span.trace_id` stays `None`. `True` sends synchronously - on a root span the whole tree is drained first - so `span.trace_id` is populated once the block exits |
| `monitor` | `bool \| None` | Three states. `True`: check this trace against [Monitor](/sdk/monitor) patterns immediately. `False`: opt this trace out of **every** ingest-time check (patterns, online evaluators, topics) - use it for traces produced inside evaluation runs, whose dataset judge already scores them, so nothing gets judged twice. `None` (default): the engine's normal behavior. |
| `pattern_ids` | `list[str]` | Restricts detection to these pattern ids when `monitor=True`. Omit to run the full default sweep (built-ins plus every active pattern) instead. |
| `span_kind` | `str` | What kind of step this span is: `"agent"`, `"llm"`, `"tool"`, `"retrieval"`, `"chain"`, `"embedding"`, `"reranker"`, `"guardrail"`, `"evaluator"`, `"prompt"`, `"memory"`. Also accepted on `child_span()`. Optional - a span that says nothing is classified by the engine's fallback rules. See [Span Kinds](/trace/span-kinds). |
## Framework examples
The decorator shown here records one trace per call - simple and dependable. For the full
execution tree (graph nodes, every LLM call, every tool call as its own timed span), use the
per-framework [integrations](/sdk/integrations/langchain) instead: a LangChain/LangGraph
callback handler, an OpenAI Agents trace processor, a CrewAI/AutoGen/ADK hook, or
[MLflow OTLP export](/sdk/integrations/databricks). Both approaches share the same tracer and
the same dashboard views.
```python LangChain theme={null}
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
from agentx import AgentX
client = AgentX.from_env()
llm = ChatOpenAI(model="gpt-4o")
@client.tracer.trace("support-agent", framework="langchain", model="gpt-4o")
def handle(query: str) -> str:
return llm.invoke([HumanMessage(content=query)]).content
handle("How do I reset my password?")
client.tracer.flush(timeout=10)
```
```python CrewAI theme={null}
from crewai import Agent, Task, Crew
from agentx import AgentX
client = AgentX.from_env()
@client.tracer.trace("research-crew", framework="crewai")
def run_crew(topic: str) -> str:
agent = Agent(role="Researcher", goal=f"Research {topic}", backstory="...")
task = Task(description=f"Research {topic}", agent=agent, expected_output="...")
crew = Crew(agents=[agent], tasks=[task])
return str(crew.kickoff())
run_crew("What is the refund policy?")
client.tracer.flush(timeout=10)
```
```python OpenAI Agents SDK theme={null}
import asyncio
from agents import Agent, Runner
from agentx import AgentX
client = AgentX.from_env()
agent = Agent(name="Support", instructions="You are a helpful support agent.")
@client.tracer.trace("openai-support-agent", framework="openai-agents", model="gpt-4o")
async def run(query: str) -> str:
result = await Runner.run(agent, query)
return result.final_output
asyncio.run(run("How do I cancel my subscription?"))
client.tracer.flush(timeout=10)
```
```python Anthropic theme={null}
import anthropic
from agentx import AgentX
agentx_client = AgentX.from_env()
claude_client = anthropic.Anthropic()
@agentx_client.tracer.trace(
"claude-agent", framework="anthropic", model="claude-sonnet-4-6"
)
def run(query: str) -> str:
msg = claude_client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": query}],
)
return msg.content[0].text
run("How do I cancel my subscription?")
agentx_client.tracer.flush(timeout=10)
```
```python Google ADK theme={null}
import asyncio
from google.adk.agents import Agent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types
from agentx import AgentX
client = AgentX.from_env()
agent = Agent(
name="support_agent",
model="gemini-3.5-flash",
instruction="You are a helpful support agent.",
)
runner = Runner(
agent=agent,
app_name="support-app",
session_service=InMemorySessionService(),
)
@client.tracer.trace("google-support-agent", framework="google-adk", model="gemini-3.5-flash")
async def run(query: str) -> str:
session = runner.session_service.create_session_sync(
app_name="support-app", user_id="user-1"
)
for event in runner.run(
user_id="user-1",
session_id=session.id,
new_message=types.Content(role="user", parts=[types.Part(text=query)]),
):
if event.is_final_response():
return event.content.parts[0].text
return ""
asyncio.run(run("How do I cancel my subscription?"))
client.tracer.flush(timeout=10)
```
## Session grouping
Use `session_id` to link traces from the same user conversation:
```python theme={null}
import uuid
session_id = str(uuid.uuid4())
@tracer.trace("support-agent", session_id=session_id)
def handle(query: str) -> str:
...
handle("First question")
handle("Follow-up question")
# Both traces appear linked in the UI
```
Traces sent without a `session_id` each get their own auto-generated session, so passing it is only about grouping - never required. On [self-host](/trace/sessions), sessions are a first-class surface: a Sessions view lists each conversation with turn counts and a conversation-level coherence score, and online evaluators can judge whole sessions (`scope="session"`) once they go idle.
## Error handling
Exceptions inside a traced function are captured as the `error` field and re-raised. The trace is still submitted:
```python theme={null}
@tracer.trace("my-agent")
def risky(query: str) -> str:
raise ValueError("Model unavailable")
try:
risky("test")
except ValueError:
pass
# Trace submitted with error: "Model unavailable"
```
To set an error manually in context manager mode:
```python theme={null}
with tracer.trace("my-agent") as span:
try:
result = call_agent(query)
span.output = result
except Exception as e:
span.set_error(str(e))
raise
```
## Flushing
Traces are sent in a background thread. Call `flush()` before process exit in scripts:
```python theme={null}
tracer.flush(timeout=5.0)
```
## Evaluating a trace
Score a previously recorded trace against a dataset without re-running the agent:
```python theme={null}
result = tracer.evaluate_trace(
trace_id="6876abc123def456789abc01",
dataset_id="6876ddd222bbb333ccc444ee",
question_index=0,
)
print(result["rating"]) # 0–10
print(result["justification"]) # LLM explanation
```
This is for scoring a trace you already recorded, standalone. To capture a trace *while* an [evaluation](/sdk/evaluations/overview) run is scoring your agent, so the run's own results show a full Execution Timeline and not just a rating, see the next section instead.
## Linking a trace to an evaluation result
[`client.evaluations.run(...).execute(your_fn)`](/sdk/evaluations/quickstart) calls `your_fn` once per test case and scores whatever it returns. Wrap the call inside `your_fn` with `tracer.trace(..., sync=True)` and return `span.trace_id` alongside your output. The eval result gets linked to the real trace, so its row in the dashboard shows a "View trace" action opening the full Execution Timeline, not just the score:
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
def support_agent(case):
with client.tracer.trace(
"support-agent-call",
input={"query": case.query},
framework="openai",
model="gpt-4o-mini",
sync=True, # required, the default async mode never returns a trace_id
monitor=False, # the run's judge already scores this case - skip ingest-time judging
) as span:
response = call_llm(case.query)
span.output = response
return {
"output": response,
"trace_id": span.trace_id, # <- links this result to the trace above
}
report = (
client.evaluations
.run(dataset_id="...", subject={"kind": "custom_agent", "framework": "openai"})
.execute(support_agent)
.finalize()
.analyze()
)
```
`trace_id` is a plain top-level key in the dict your function returns, in the same place as `output` and `metadata`. It works the same way if your function returns an `EvaluationResult` directly (set `trace_id=span.trace_id` on it) instead of a dict.
`sync=True` blocks until the trace is ingested (typically well under a second) before your function returns. For a high test-case count where that latency adds up, keep tracing async (the default) and skip the timeline for that run. You'll still get the score.
## Monitor
Every trace you send can also be checked against detection patterns and turned into a triage-ready signal, either immediately via `tracer.trace(..., monitor=True, pattern_ids=[...])` with no dashboard setup, or automatically for every trace once a monitoring profile is enabled for the agent in Governance > Agents. See [Monitor](/sdk/monitor) for the full picture.
## Limits
| Limit | Value |
| ------------------------------------ | ------------------------------------------------------------------------------------- |
| Max tool calls per trace | 50 (excess calls are silently truncated) |
| Rate limiting | None currently enforced per API key |
| `input` / `output` / `metadata` size | No application-level limit; bounded by the backing store's row limits (hosted: 16 MB) |
# TypeScript CI SDK
Source: https://developers.agentx.so/sdk/ts-eval
@agentx/eval - the minimum eval slice for a TypeScript pipeline: submit, finalize, gate
`@agentx/eval` is a zero-dependency TypeScript client for the CI slice of offline evaluation:
open a run against an existing dataset, submit your agent's outputs (the engine judge-scores
each batch synchronously), finalize, and gate the build. Node 18+, ESM and CJS, nothing else -
it uses the built-in `fetch`.
Be clear about what it is: **the CI surface only**. Dataset curation, grading configs, the
review queue, analysis reports, and everything else on these pages live in the
[Python SDK](/sdk/evaluations/overview) and the dashboard. Use this package when the repo being
gated is TypeScript and pulling in a Python step just for the gate is the only reason it would
exist.
```bash theme={null}
yarn add @agentx/eval
```
```ts theme={null}
import { AgentXEval } from "@agentx/eval";
const evals = new AgentXEval({
apiKey: process.env.AGENTX_API_KEY!,
baseUrl: "http://localhost:4700/api/v1",
});
const run = await evals.initRun({
datasetId: "existing-dataset-id",
subject: { name: "support-agent", metadata: { version: process.env.GIT_SHA } },
});
const outputs = await runMyAgentOverCases(); // your code
await run.submit(
outputs.map((o, caseIndex) => ({
caseIndex,
query: o.query,
output: o.answer,
traceId: o.traceId, // optional: links the trace for trajectory-aware judging
}))
);
const summary = await run.finalize();
console.log(summary.liveStatistics); // averageRating, ratedCount, skippedCount, failedCount
const gate = await run.gate({ failUnder: 7, noRegression: true, record: true, caller: "ci" });
gate.assert(); // throws an Error naming each failed check when the gate did not pass
```
The gate checks are the same ones the [Python gate](/integrations/self-host-ci) runs -
`failUnder` (absolute floor) and `noRegression` (tolerance default 0.5, needs run history) -
and recorded gates land in the same dashboard CI Gates history.
Two more pieces round out the CI story. Pairwise comparison against a previous run:
```ts theme={null}
const cmp = await evals.comparePairwise({ runAId: run.runId, runBId: "previous-run-id", bothOrders: true });
console.log(cmp.summary); // { total, aWins, bWins, ties, winner, flipRate }
```
And resume after an interrupted job - `submittedKeys()` returns what the engine already has,
so a re-run only submits the remainder:
```ts theme={null}
const done = new Set(await run.submittedKeys());
const remaining = allCases.filter((c, i) => !done.has(run.idempotencyKey(i)));
```
`submit()` auto-chunks at 10 results per batch and retries each batch once; a second failure
throws an `AgentXEvalError` with the HTTP status rather than finishing silently. The package
README in the `AgentX-Trace-Eval` repo (`packages/agentx-eval`) is the full reference.
# Backup & export
Source: https://developers.agentx.so/self-host/backup
Bulk NDJSON export, incremental snapshots, and the restore runbook
Your data never needs a support ticket to leave the box. Every project-scoped table streams out
of the engine as NDJSON over one authenticated endpoint, and the same endpoint powers
`client.export` in the Python SDK.
## What's exportable
`GET /api/v1/export` (with your project's `x-api-key`) returns a manifest of every entity with
live row counts. `GET /api/v1/export/` streams the rows, one JSON object per line, in
exactly the stored shape (timestamps as ISO-8601):
| Entity | Contents |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `traces` | Every trace/span row: input, output, tokens, latency, session and span ids, metadata |
| `signals`, `signal-feedback` | Triage signals and the human verdicts recorded on them |
| `events`, `classifications` | Scorer/evaluator event history and topic classifications |
| `runs`, `run-results`, `gate-results` | Evaluation runs, per-case results, CI gate verdicts |
| `pairwise-comparisons` | Head-to-head judge verdicts between runs |
| `datasets` | Evaluation datasets with their cases |
| `evaluation-settings` | Grading configs: the judge rubric + profile behind each dataset run and online evaluator |
| `dataset-versions`, `evaluation-settings-versions` | Version histories: every recorded edit to a dataset or grading config |
| `evaluation-analyses` | The AI analysis narratives generated for runs (strengths, weaknesses, recommendations) |
| `review-queue` | Human review-queue items with their labels and corrected scores |
| `feedback`, `outcomes` | End-user votes and reported real-world outcomes (ground truth) |
| `session-scores` | Whole-session judge scores |
| `patterns`, `online-evaluators`, `custom-evaluators`, `rules` | Scorer configuration: your patterns, LLM judges, code/external scorers, automation rules |
| `playground-profiles` | Saved Playground configurations |
Config tables ride along with the data because a usable backup is the data plus the scorer
configuration that produced it. Everything is scoped to the API key's project: an export can
never cross a tenant boundary.
Instance-wide state (auth users/orgs, app settings, the pricing catalog) is deliberately not in
the project export; it belongs to the database-level backup below.
## Exporting
```python Python SDK theme={null}
from agentx import AgentX
client = AgentX(api_key="agtx_...")
# Full backup: one .ndjson per entity + manifest.json
client.export.dump("./backup-2026-08-22")
# Incremental (entities filter on their own timestamp column)
client.export.dump("./nightly", since="2026-08-21T00:00:00Z")
# Stream without touching disk
for row in client.export.iter("traces"):
...
```
```bash curl theme={null}
# Manifest with row counts
curl -H "x-api-key: $KEY" http://localhost:4700/api/v1/export
# One entity, incrementally
curl -H "x-api-key: $KEY" \
"http://localhost:4700/api/v1/export/traces?since=2026-08-21" \
-o traces.ndjson
```
Exports are keyset-paginated internally, so memory stays flat on both ends regardless of table
size, and a nightly `dump(since=...)` only moves the delta.
## Restore runbook
There are two supported restore paths. There is deliberately no blind row-level import
endpoint: one would bypass the engine's invariants (span dedupe, id uniqueness, derived agent
rows) and could corrupt a live project silently.
### Path 1: database-level (full instance restore)
In the SQLite and Postgres tiers the engine owns exactly one database; restoring it restores
everything, including auth and settings.
* **SQLite** (default): stop the engine, copy `$AGENTX_HOME/agentx.db` back into place, start
the engine. For hot backups use `sqlite3 agentx.db ".backup backup.db"` which is safe while
the engine runs.
* **Postgres**: standard `pg_dump` / `pg_restore` (or your provider's point-in-time recovery).
The engine runs its own migrations at boot, so restoring an older dump into a newer engine is
supported; the reverse is not.
* **Enterprise tier** (Postgres + ClickHouse): the control plane is the Postgres backup above;
spans live in ClickHouse and are append-only, so incremental strategies work well - a native
format dump (`SELECT * FROM agentx_spans FORMAT Native`), `clickhouse-backup`, or volume
snapshots. Restore into a fresh table (the engine creates it on boot) with
`INSERT INTO agentx_spans FORMAT Native`.
### Path 2: replay (project-level, cross-instance migration)
NDJSON exports replay through the normal ingest surface into any project on any instance:
```python theme={null}
import json
from agentx import AgentX
target = AgentX(api_key="agtx_target_project_key")
with open("backup/traces.ndjson") as fh:
for line in fh:
row = json.loads(line)
with target.tracer.trace(
row["name"], input=row["input"],
session_id=row.get("sessionId"), metadata=row.get("metadata"),
) as span:
span.output = row["output"]
```
Replay is how the engine's own round-trip test verifies the export contract: export a seeded
project, replay it into a fresh one, and the counts and contents match. Ground truth
(`feedback`, `outcomes`) replays the same way through `client.feedback` / `client.outcomes`.
Note what replay preserves and what it does not: content, sessions, span trees, and metadata
survive; engine-assigned row ids and `createdAt` are newly assigned on the target (the original
timestamps remain inside the exported file if you need them).
### Dataset import & delete
Datasets get a first-class replay path of their own: `POST /api/v1/evaluate/datasets/import`
takes a row from the `datasets` NDJSON export as-is and recreates it - cases, grading config,
similarity flags - as a **new** dataset with fresh ids (201). It is a copy, never an in-place
restore, so importing can't clobber a live dataset:
```bash theme={null}
head -1 backup/datasets.ndjson | \
curl -X POST http://localhost:4700/api/v1/evaluate/datasets/import \
-H "x-api-key: $KEY" -H "Content-Type: application/json" -d @-
```
The reverse also exists now: datasets can be **deleted from the dashboard**. Deleting removes
the dataset, its grading config, and both version histories; past runs are kept - run history
is your record of what was measured, not part of the dataset. A dataset whose grading config is
still bound to a live online evaluator refuses deletion (409) until the scorer is detached.
## Suggested schedule
* **Nightly**: `client.export.dump(dir, since=<24h ago>)` to object storage - the incremental
NDJSON is your audit-friendly, vendor-neutral copy.
* **Weekly**: database-level backup (SQLite `.backup` file or `pg_dump`) - the fast full-restore
path.
* **Before upgrades**: database-level backup, always; the engine migrates forward only.
# Configuration
Source: https://developers.agentx.so/self-host/configuration
Every flag and environment variable with its default, deployment tiers, Prometheus metrics, the audit trail, and the end-to-end smoke test
The engine reads all configuration from CLI flags and environment variables - there is no
config file to edit. Everything has a working default: a bare `agentx-server` boots SQLite on
port 4700 with no keys and no auth.
## Core
| Variable / flag | Default | Description |
| -------------------------------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--dev` | off | Opens the dashboard in your browser on startup. The API behaves identically without it. |
| `--upgrade` | off | Re-downloads the latest dashboard bundle before serving, even when one already exists. See [Upgrading](/self-host/installation#upgrading). |
| `--port` / `PORT` | `4700` | Port for the engine's HTTP API (the dashboard is served on the same port). |
| `--db-url` / `AGENTX_DB_URL` | local SQLite | Postgres connection string, e.g. `postgres://user:pass@host:5432/db`. |
| `--engine-bin` | auto-detected | Overrides which engine executable `agentx-server` launches. |
| `AGENTX_HOME` | `~/.agentx` | Where the SQLite file (`agentx.db`) and `config.json` live. |
| `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY` / `OPENROUTER_API_KEY` | unset | Unlock the LLM-judge features (evaluation scoring, online evaluators, semantic patterns, suggestion drafting, Playground). Also settable live from Platform Settings. Trace ingest and phrase/regex patterns need none. |
| `AGENTX_LOG_LEVEL` | `info` | Log level for the engine's structured JSON log lines on stdout. Pipe through `pino-pretty` for colors in dev. |
## Deployment tiers
One engine binary, one wire API - the tier is a deployment choice, never a fork:
| Tier | Control plane | Telemetry (spans) | Configure with |
| ---------- | ------------- | ----------------- | ------------------------------------------------------------------------ |
| Self-host | SQLite | SQLite, same file | nothing - the default |
| Team | Postgres | Postgres | `AGENTX_DB_URL=postgres://...` |
| Enterprise | Postgres | ClickHouse | `AGENTX_DB_URL=...` + `AGENTX_TELEMETRY_URL=http://user:pass@ch:8123/db` |
| Variable | Default | Description |
| ------------------------------------ | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTX_TELEMETRY_URL` | unset | ClickHouse URL for span storage (the enterprise tier). Unset keeps spans in the same SQLite/Postgres database as everything else. The boot log confirms with `Telemetry store: ClickHouse (...)`. |
| `AGENTX_TELEMETRY_TTL_DAYS` | `90` | ClickHouse span retention, baked into the table's TTL on first boot. Changing it later needs `ALTER TABLE agentx_spans MODIFY TTL`. |
| `AGENTX_PG_PARTITION_RETENTION_DAYS` | unset (keep everything) | Partitioned-Postgres tier: whole daily partitions older than this are dropped by the daily maintenance timer. Fresh Postgres installs get a natively partitioned traces table; pre-existing databases keep their non-partitioned table and row-delete retention (detected automatically). |
The repo ships an enterprise reference deployment as `docker-compose.enterprise.yml`
(Postgres + ClickHouse + one engine) and a Helm chart at `deploy/helm/agentx` (build and push
the repo `Dockerfile` first - no public image is published yet; `postgres.externalUrl` /
`clickhouse.externalUrl` point it at managed databases). See
[High availability & DR](/self-host/high-availability) for the topology rules and
`engine/docs/deployment-tiers.md` in the repo for the full walkthrough, including how to verify
spans actually land in ClickHouse.
## Using Postgres
```bash theme={null}
AGENTX_DB_URL=postgres://user:pass@host:5432/agentx agentx-server --dev
```
Tables are created and migrated automatically at boot - no separate migration step. SQLite
(default) is a real, durable option for a single-machine install; reach for Postgres when you
want managed backups, point-in-time recovery, or higher retention.
## Ingest and rate limits
| Variable | Default | Description |
| ------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `AGENTX_INGEST_FLUSH_MS` | `10` | Micro-batch window: ingest requests arriving inside it share one INSERT. |
| `AGENTX_INGEST_FLUSH_SIZE` | `500` | Early-flush batch size. |
| `AGENTX_INGEST_QUEUE_MAX` | `10000` | Bound on the ingest queue; beyond it ingest sheds load with 429 + `Retry-After` (SDKs back off and redeliver; span ids make retries idempotent). |
| `AGENTX_INGEST_MAX_FIELD_CHARS` | `100000` | Per-field payload cap (serialized length). Oversized fields are truncated with an explicit marker. |
| `AGENTX_RATE_LIMIT_CREDENTIAL` | `120`/min | Request ceiling per IP for the credential surface (sign-in, anything that hands out or is guarded by a key). |
| `AGENTX_RATE_LIMIT_DATA_PLANE` | `6000`/min | Request ceiling per IP for the data plane - far above any real SDK burst; exists only to bound key-guessing loops. |
| `AGENTX_RATE_LIMIT` | on | `off` disables rate limiting entirely (benchmarks, or when the load balancer rate-limits instead). Counters are per-process and per-IP. |
## Monitoring behavior
| Variable | Default | Description |
| ----------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTX_MONITOR_CHILD_SPANS` | `false` | Online scoring runs on root spans only (scoring a tool call's output as if it were the whole interaction is misleading, and it multiplies judge calls per trace). `true` restores per-span scoring. |
| `AGENTX_OTEL_MONITOR` | `true` | Traces arriving via the OTLP endpoint (`/api/v1/otel/v1/traces`) are monitored by default - pointing an OTel exporter here is itself the opt-in. `false` disables. |
| `AGENTX_SESSION_SWEEP` | `true` | `false` disables the background sweep that judges idle sessions for session-scoped online evaluators (`POST /api/v1/agent-monitoring/session-sweep/run` still works on demand). |
| `AGENTX_SESSION_SWEEP_WINDOW_HOURS` | `24` | How far back the session sweep looks for judgeable sessions. Bucketed: up to 24 means 24h, up to 168 means 7d, above that 30d. |
| `AGENTX_IMPROVEMENT_SWEEP` | `true` | `false` disables the background sweep that drafts and validates improvement proposals for the [Improvement Inbox](/improve/validating-proposals#the-improvement-inbox) (`POST /api/v1/evaluate/improve/inbox/sweep/run` still works on demand). |
## Quotas
| Variable | Default | Description |
| ----------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTX_QUOTA_JUDGE_CALLS_PER_DAY` | unset (unlimited) | Daily cap on judge LLM calls - per organization in multi-tenant mode, per instance otherwise. |
| `AGENTX_QUOTA_ONLINE_JUDGE_CALLS_PER_DAY` | unset (unlimited) | Daily per-project cap on live trace-scope judge calls specifically ([online evaluators](/monitor/online-evaluators)), on top of the global judge quota. |
| `AGENTX_QUOTA_TRACES_PER_DAY` | unset (unlimited) | Daily cap on ingested root traces, per project. Child spans of existing traces don't count. |
## Auth and multi-user
The default posture is auth-disabled: a reachable port is a trusted user, and the dashboard
connects itself. Everything below only matters once you turn on real accounts - see
[Authentication](/authentication#the-three-modes) for the full model.
| Variable | Default | Description |
| ------------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `AGENTX_AUTH` | `disabled` | `enabled` turns on real user accounts for the dashboard - owner setup on first boot, sessions, org-scoped project keys. See [Authentication](/authentication#the-three-modes). |
| `AGENTX_AUTH_SECRET` | auto-generated | Session-signing secret. Unset, one is generated on first enabled boot and persisted in the database, so sessions survive restarts with no setup step. Set it explicitly when several environments must share one secret. |
| `AGENTX_MULTI_TENANT` | `false` | With auth enabled: every signup gets its own organization + seeded project, per-org LLM keys and pricing catalog, invitations via Settings → Team. The SaaS/multi-tenant posture - see [Authentication](/authentication#the-three-modes). |
| `AGENTX_PUBLIC_URL` | unset | The externally-reachable base URL: cookie/redirect correctness behind a proxy in enabled mode, the base for password-reset and verification links (REQUIRED on any mail-sending deployment - without it those links derive from the request's Host header, which a client can forge; the engine warns at boot), and the base for the [MCP OAuth callback](/improve/mcp). |
| `AGENTX_TRUSTED_ORIGINS` | unset | Comma-separated extra origins allowed to hit the auth routes in enabled mode. |
| `AGENTX_OPEN_SIGNUP` | unset | Enabled mode only. By default, signing up grants NO organization membership once the org has a member - teammates join by accepting an invitation (Settings > Team), so an exposed port cannot hand your projects to a stranger who self-registers. `true` restores auto-join for closed networks. |
| `AGENTX_TRUST_PROXY` | unset | Behind a reverse proxy, set to `true` (or a hop count / CIDR, Express `trust proxy` semantics) so rate limits and audit rows see the real client IP instead of the proxy's. Leave unset on a directly-exposed port - trusting the header there lets clients spoof their IP. |
| `AGENTX_ADMIN_TOKEN` | unset | Unlocks the operator endpoints (send as `x-admin-token`): `GET /api/v1/admin/overview` (per-org members, projects, 24h judge/trace usage) and `GET /api/v1/admin/audit` (the instance-wide [audit trail](#the-audit-trail)). Unset = the endpoints don't exist. |
### Email
| Variable | Default | Description |
| ----------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTX_RESEND_API_KEY` | unset | Sends auth email (invitations, password reset, verification) through [Resend](https://resend.com). Configuring any mail transport turns on the dashboard's "Forgot password?" flow. |
| `AGENTX_SMTP_URL` | unset | Alternative mail transport: an SMTP connection URL, e.g. `smtp://user:pass@smtp.example.com:587`. |
| `AGENTX_EMAIL_FROM` | `AgentX ` | From address for auth email. |
| `AGENTX_EMAIL_DEBUG_DIR` | unset | Dev-only mail transport: writes each email as a JSON file into this directory instead of sending it. |
| `AGENTX_REQUIRE_EMAIL_VERIFICATION` | `false` | With a mail transport configured: new accounts must click a verification link before signing in. |
### Social sign-in and SSO
| Variable | Default | Description |
| ---------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTX_GOOGLE_CLIENT_ID` / `AGENTX_GOOGLE_CLIENT_SECRET` | unset | Enables "Continue with Google" on the sign-in screen. OAuth callback: `/api/v1/auth/callback/google`. |
| `AGENTX_GITHUB_CLIENT_ID` / `AGENTX_GITHUB_CLIENT_SECRET` | unset | Enables "Continue with GitHub". OAuth callback: `/api/v1/auth/callback/github`. |
| `AGENTX_OIDC_ISSUER` / `AGENTX_OIDC_CLIENT_ID` / `AGENTX_OIDC_CLIENT_SECRET` | unset | Generic OIDC SSO: point the trio at any IdP that serves `/.well-known/openid-configuration` (Okta, Entra ID, Auth0, Google Workspace, Keycloak, ...) and an SSO button appears on the sign-in screen. Callback URL to register with the IdP: `/api/v1/auth/callback/oidc` (engines older than better-auth 1.7 used `/api/v1/auth/oauth2/callback/oidc`; update the IdP redirect URI when upgrading). SAML and SCIM are not supported. |
| `AGENTX_OIDC_NAME` | `SSO` | The SSO button's label, e.g. `Okta`. Cosmetic only. |
## Wire casing
The API's one wire convention is **camelCase** (`sessionId`, `latencyMs`, `judgeScorers`).
The trace-ingest write path historically accepted snake\_case keys (`session_id`,
`latency_ms`, ...); those remain accepted as legacy aliases for existing SDKs, and
`POST /ingest/traces` answers with both `traceId` (canonical) and `trace_id` (legacy). New
endpoints and fields are camelCase only.
## The project API key
The startup log's `Default project API key: agtx_local_...` is generated once on first run and
reused after that (persisted in `$AGENTX_HOME/config.json`). It's what SDK/CI/OTel callers send
as `x-api-key`. In the default auth-disabled mode the dashboard fetches it automatically from
`GET /api/v1/auth/config`, so there is nothing to paste. Additional projects (created from the
project switcher) each get their own key, visible in that project's **Settings → API access**.
## Prometheus metrics
`GET /metrics` serves Prometheus text format on the engine's one port - open by default (the
content is deliberately operational-only: queue counters, RSS, uptime; never span content,
keys, or per-project data). Internet-exposed deployments set `AGENTX_METRICS_TOKEN` and scrape
with `Authorization: Bearer `:
```bash theme={null}
curl -H "Authorization: Bearer $AGENTX_METRICS_TOKEN" http://localhost:4700/metrics
```
| Metric | Type | Meaning |
| ------------------------------------------------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `agentx_ingest_queue_depth` / `agentx_ingest_queue_max_depth` | gauge | Spans currently queued for flush, and the high-water mark. |
| `agentx_ingest_spans_total` / `agentx_ingest_spans_stored_total` | counter | Spans accepted into the queue / durably stored. |
| `agentx_ingest_spans_deduped_total` | counter | Replays deduped by the idempotency key. |
| `agentx_ingest_spans_rejected_total` | counter | Spans shed with 429 (queue full). |
| `agentx_ingest_spans_dropped_total` | counter | Spans lost after a failed batch retry - **should be 0**; above zero is an incident (see the repo's `engine/docs/runbook.md`). |
| `agentx_ingest_batches_total` | counter | Flush batches executed. |
| `agentx_process_resident_memory_bytes` / `agentx_process_uptime_seconds` | gauge | Process RSS and uptime. |
## The audit trail
The engine keeps an append-only audit log of control-plane activity: scorer/judge/pattern
create-update-delete, settings changes, API key regenerations, project create/delete, bulk
export reads, and (in `AGENTX_AUTH=enabled` mode) sign-in/sign-up/sign-out attempts with the
attempted email and status. Data-plane traffic (trace ingest, feedback, outcomes) is
deliberately excluded so the trail stays readable, and recorded summaries carry field *names*
only, never values - scripts, keys, and passwords cannot leak into the log.
```bash theme={null}
# Instance-wide, operator token required (AGENTX_ADMIN_TOKEN)
curl -H "x-admin-token: $TOKEN" \
"http://localhost:4700/api/v1/admin/audit?since=2026-08-01&action=scorer.create&limit=100"
# Org-scoped, session cookie (enabled mode): your own organizations' projects only
curl -b "$SESSION" http://localhost:4700/api/v1/auth-org/audit
```
Rows are immutable by construction: the engine contains an insert and a read for
`audit_events` and nothing else - no update or delete surface exists.
## Verifying end-to-end
The repo ships a smoke test that runs the real Python SDK against a fresh instance - Trace,
Evaluate (judge scoring), and Monitor (built-in + custom pattern detection). It needs an
`OPENAI_API_KEY` (for the judge paths) and Python 3:
```bash theme={null}
OPENAI_API_KEY=sk-... ./scripts/smoke-test.sh
# Or against a throwaway Postgres instead of SQLite:
OPENAI_API_KEY=sk-... AGENTX_DB_URL=postgres://postgres:agentx@localhost:55432/agentx \
./scripts/smoke-test.sh
```
## OpenRouter
Save an OpenRouter API key (**Platform Settings → LLM Providers**, or `OPENROUTER_API_KEY` in
the environment) and every model field in the product - the platform model picker, judge
scorers, the playground - can call the full OpenRouter catalog by its `vendor/model` ids
(`anthropic/claude-sonnet-4.5`, `meta-llama/llama-4-70b`, ...). Any model id containing `/`
routes through OpenRouter's OpenAI-compatible API with that key; the model picker lists the
live catalog once the key is configured.
## Platform model
The engine runs its own LLM operations - topic mapping, suggestions and answer drafts, dataset
coverage analysis, prompt/tool proposals, AI analysis, auto-improve reports - on a single
default model (`gpt-5.6-luna` out of the box). **Platform Settings → LLM Providers → Platform
model** changes it instance-wide via a searchable picker: direct OpenAI/Anthropic/Gemini
names, custom endpoints from the pricing catalog, and - with an OpenRouter key - the whole
OpenRouter catalog. This is deliberately separate from judge scorers, which configure their
model per scorer; scorers left without a model keep the built-in judge default.
# High availability & DR
Source: https://developers.agentx.so/self-host/high-availability
The single-writer topology, scaling by tier, zero-downtime-ish upgrades, and recovery objectives
The engine is one near-stateless binary in front of your database(s). That shape makes the
availability story short: make the **database** highly available like any Postgres, keep the
engine restartable in seconds, and know the one hard topology rule below.
## The topology rule: one engine instance
The engine runs as a **single instance in every tier** - it is the one telemetry writer. The
ingest queue micro-batches and dedupes per process, and the ClickHouse adapter enforces span
idempotency adapter-side on the same assumption; the Helm chart pins `replicas: 1` with a
`Recreate` strategy for exactly this reason. Multi-writer ingest is a named future ADR, not a
`--set replicas=3` away.
What you scale instead is the storage tier:
| Tier | Configure with | What it buys |
| --------------------- | ------------------------------------------- | ----------------------------------------------------------------------------------- |
| SQLite (default) | nothing | A laptop or single VM; durable, WAL-mode, crash-safe |
| Postgres | `AGENTX_DB_URL=postgres://...` | Managed backups, PITR, replicas, partitioned traces table with drop-based retention |
| Postgres + ClickHouse | + `AGENTX_TELEMETRY_URL=http://...:8123/db` | Columnar span storage: flat query latency at high retention, TTL-based retention |
A single engine process is not the bottleneck it sounds like: the measured baseline
(`engine/scripts/bench.mjs`, 32 concurrent senders, production-shaped payloads) sustains
**\~2,300 stored spans/s with an ingest ack p95 of 18 ms and dashboard query p95 of 8 ms** - at
that concurrency the HTTP layer, not storage, is the bound. Judge features are bounded by your
LLM provider, not the engine. CI re-measures this nightly with regression floors, and a weekly
chaos suite drills the failure modes in the repo's `engine/docs/runbook.md`.
## What makes restarts and redeploys safe by design
Availability for a single-instance engine means fast, safe restarts, and the engine is built
for them:
* **Graceful shutdown**: SIGTERM/SIGINT stop accepting connections, drain the ingest queue,
flush, and close the database - including signals that land mid-boot.
* **Idempotent migrations** run at boot (`IF NOT EXISTS` + additive backfills), so a restart
or a brief old/new overlap during a redeploy never races destructively.
* **Lease-protected background sweeps** (idle-session judging, improvement proposals) take a
database lease before each tick, so an overlapping old and new process during a handoff
never judges the same session twice on your API key (`sweepLease.test.ts`, plus a Postgres
variant, prove it with two lease holders against one database).
* **Client-side buffering**: the SDK queues and retries with backoff on 429/503, and span ids
make redelivery idempotent - a restart window loses nothing that a client was still holding.
## Upgrades
Engines migrate the schema forward at boot and never backward. The safe order:
1. Database-level backup (see [Backup & export](/self-host/backup) - always before upgrades).
2. Stop the old engine, start the new one, wait for `GET /health`. With `Recreate` semantics
this is a seconds-long gap; SDKs ride it out on their retry queue.
3. Roll back = restore the pre-upgrade database backup + the old binary. A newer database
under an older engine is not supported.
## RPO / RTO
| Objective | With nightly NDJSON + weekly pg\_dump | With Postgres PITR/streaming replica |
| ---------------------- | ------------------------------------------ | ---------------------------------------------------- |
| RPO (data loss window) | up to 24h | seconds |
| RTO (time to restore) | minutes-hours (restore dump, replay delta) | minutes (promote replica, restart engine against it) |
The pragmatic middle ground for most teams: managed Postgres with point-in-time recovery
turned on, plus the nightly `client.export.dump(since=...)` NDJSON snapshot to object storage
as the vendor-neutral copy that survives even a bad database migration. In the ClickHouse tier,
add span backups per [Backup & export](/self-host/backup#path-1-database-level-full-instance-restore) -
spans are append-only, so incrementals are cheap.
## Monitoring the engine itself
* `GET /health` returns `{"status":"ok"}` - point your orchestrator's liveness/health check
here (it sits under the data-plane rate ceiling, far above any prober).
* `GET /metrics` serves [Prometheus metrics](/self-host/configuration#prometheus-metrics):
queue depth, stored/deduped/rejected/dropped counters, RSS, uptime.
`agentx_ingest_spans_dropped_total` above zero is an incident; sustained 429s
(`agentx_ingest_spans_rejected_total` climbing) mean the storage backend cannot keep up -
raising `AGENTX_INGEST_QUEUE_MAX` buys burst absorption, not throughput.
* Ingest latency is the canary metric: it is a micro-batched insert, so a rising ack p95 means
database pressure before anything else notices.
* If ClickHouse goes down (enterprise tier), ingest answers 503 + `Retry-After` and clients
redeliver; the control plane and dashboard stay up, and ingestion resumes on recovery with
no operator action.
* The [audit trail](/self-host/configuration#the-audit-trail) records config changes and bulk
exports into the database, so the trail survives engine restarts and redeploys.
## Rate limits behind a load balancer
Rate-limit counters are in-memory and per-IP. Behind a proxy or load balancer, either
terminate rate limiting there and set `AGENTX_RATE_LIMIT=off` inside, or make sure the engine
sees real client IPs - otherwise every client shares the proxy's counter.
# Installation
Source: https://developers.agentx.so/self-host/installation
curl | bash, the Python SDK launcher, Docker, or build from source - then connect the dashboard and SDK
Prebuilt releases cover macOS and Linux on arm64/amd64 - no Go, Node, or Bun needed at runtime.
Pick one path:
| Path | Best for |
| ---------------------------------------- | ------------------------------------- |
| [Quick install](#quick-install-prebuilt) | Trying it out, local development |
| [Docker](#docker) | Servers, teams, anything long-running |
| [From source](#build-from-source) | Contributing, patching the engine |
## Quick install (prebuilt)
```bash curl | bash theme={null}
curl -sSL https://raw.githubusercontent.com/AgentX-ai/AgentX-trace-eval/main/install.sh | bash
agentx-server --dev
```
```bash Python SDK theme={null}
pip install agentx-python
agentx-trace-eval --dev
```
Both download the same prebuilt release (engine, CLI, dashboard) into `~/.agentx/bin`
(override with `AGENTX_INSTALL_DIR`). Once it's up:
```
AgentX self-host engine listening on http://localhost:4700
Default project API key: agtx_local_...
```
`--dev` opens the dashboard in your browser; omit it to run headless (behind a process manager,
for example) - the API comes up identically either way.
## Docker
```bash theme={null}
git clone https://github.com/AgentX-ai/AgentX-trace-eval.git && cd AgentX-trace-eval
docker build -t agentx-selfhost .
docker run -d -p 4700:4700 -v agentx-data:/data agentx-selfhost
docker logs $(docker ps -lq) 2>&1 | grep "API key"
```
Open [http://localhost:4700](http://localhost:4700) - the dashboard connects itself (in the default auth-disabled mode
the engine hands the browser the Default project's API key, nothing to paste). The key from
`docker logs` is what your SDK/scripts use as `AGENTX_API_KEY`; it prints on every start. For
multi-user or network-exposed deployments set `-e AGENTX_AUTH=enabled`, which requires sign-in
and never hands the key out.
* `/data` holds the SQLite database and config (`AGENTX_HOME`) - mount a named volume so state
survives recreation, or set `AGENTX_DB_URL` to your Postgres and skip the volume.
* Pass provider keys with `-e OPENAI_API_KEY=...` (or set them later in Platform Settings).
* The image has a `/health` HEALTHCHECK.
* The build pulls the latest dashboard bundle and revalidates it on every rebuild (ETag-checked
`ADD` layer), so a newly published dashboard is picked up with no `--no-cache`. Pin one with
`--build-arg AGENTX_WEB_URL=.../releases/download/vX.Y.Z/agentx-web.tar.gz`.
The repo also ships two Compose files: `docker-compose.yml` (engine + Postgres) and
`docker-compose.enterprise.yml` (engine + Postgres + ClickHouse - the
[enterprise tier](/self-host/configuration#deployment-tiers)), plus a Helm chart at
`deploy/helm/agentx`.
## Build from source
**Prerequisites:** Node.js + [Yarn](https://yarnpkg.com/); [Go](https://go.dev/) and
[Bun](https://bun.sh/) only for the compiled single-binary distribution, not day-to-day dev.
```bash theme={null}
git clone https://github.com/AgentX-ai/AgentX-trace-eval.git && cd AgentX-trace-eval/engine
yarn && yarn dev --dev
```
That's the whole dev loop: `yarn dev` builds the `@agentx/judge-core` workspace package
automatically (a \~1s step), `.env` is optional, and if the repo-root `web/` is missing dev mode
downloads the prebuilt dashboard bundle from the repo's releases on first boot. An existing
`web/` is never touched on a normal boot - see [Upgrading](#upgrading) for refreshing it.
Offline, fetch it manually (into the **repo root**, not `engine/` - the repo-root `web/` is the
one canonical bundle location for a checkout, and a copy under `engine/web` is ignored):
```bash theme={null}
mkdir -p web && curl -fsSL https://github.com/AgentX-ai/AgentX-trace-eval/releases/latest/download/agentx-web.tar.gz | tar -xz -C web
```
For the full packaged layout (compiled engine binary + Go CLI, exactly what a release install
looks like):
```bash theme={null}
./build.sh
./dist/agentx-server --dev
```
## Upgrading
The engine and the dashboard version independently (Platform Settings shows both in its
lower-right corner, e.g. `Engine v0.3.8 · UI v0.3.6`; a source checkout reports `dev`).
**Dashboard.** A downloaded dashboard bundle is never touched on a normal boot - without this,
a source host quietly serves whatever it downloaded first, forever. Pass `--upgrade` to
re-download the latest dashboard release before serving; it works the same in every layout:
```bash theme={null}
yarn dev --upgrade # source checkout (from engine/)
agentx-server --upgrade # curl | bash install
agentx-trace-eval --upgrade # Python SDK launcher
```
The new bundle is staged and verified before it replaces the old one, so a failed or
interrupted download keeps the existing dashboard serving and logs a warning - `--upgrade` can
never break a working UI. The boot log states what happened either way
(`Upgrading the dashboard bundle in ... (currently built ...)`).
**Engine.** Re-run the installer for your path: `curl | bash` again (installs the latest
release into `~/.agentx/bin`; pin a release with `AGENTX_VERSION=vX.Y.Z`), or
`agentx-trace-eval --update` for the SDK launcher (its own flag, consumed before the handoff:
re-downloads the engine release). A source checkout upgrades with `git pull`.
Both at once, on the launcher: `agentx-trace-eval --update --upgrade` refreshes the engine and
then the dashboard.
## Connect
The dashboard connects itself: in the default auth-disabled mode the engine hands the browser
the `Default project` API key on first visit, so you land straight on a working screen. (A
connect screen only appears against an older engine that doesn't hand the key out.) Because
anyone who can reach the port gets the key, disabled mode is for local/trusted use - for a
shared, multi-user, or network-exposed instance, use
[`AGENTX_AUTH=enabled`](/authentication#the-three-modes), which requires
sign-in and never hands the key out.
Point the SDK at the engine with the same key - no separate SDK, no code changes:
```bash theme={null}
export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
export AGENTX_API_KEY=agtx_local_... # from the startup log
```
```python theme={null}
from agentx import AgentX
client = AgentX.from_env()
with client.tracer.trace("my-agent") as span:
span.input = "hello"
span.output = "hi there"
```
Everything under [Tracing](/sdk/tracing), [Monitor](/sdk/monitor), and
[Evaluations](/sdk/evaluations/overview) works the same against self-host as against the hosted
API.
## The `agentx-trace-eval` launcher
`agentx-trace-eval` (bundled with `agentx-python`) is a thin launcher, not a reimplementation:
the first run downloads the engine/CLI release into `~/.agentx/bin` and hands off to it, so
installing the Python SDK stays light for the common hosted-API case.
* Each SDK release pins the engine release it was tested against, and the launcher converges
the install to that pin - upgrading the SDK upgrades the engine to the matching pair.
* `AGENTX_TRACE_EVAL_VERSION` overrides the pin with a specific tag, or `latest` (which trusts
whatever is installed and prints a notice when a newer release exists).
* `--update` is the launcher's own flag (force-reinstall the resolved engine release); every
other flag - `--dev`, `--upgrade`, `--port`, `--db-url` - passes through to `agentx-server`
untouched. See [Upgrading](#upgrading).
* `AGENTX_TRACE_EVAL_SKIP_WEB` skips downloading the dashboard bundle for headless use (CI, for
example).
There is no Homebrew formula - `curl | bash`, the SDK launcher, and Docker are the supported
paths.
# Self-Host Overview
Source: https://developers.agentx.so/self-host/overview
The full Trace, Monitor, Evaluate, and Improve stack on your own machine - no account, bring your own LLM keys
[AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-Trace-Eval) is the open-source,
self-hosted build of AgentX's governance layer. One local install gives you the complete loop -
**Trace**, **Monitor**, **Evaluate**, **Improve** - with no account and no multi-tenant billing.
You bring your own OpenAI, Anthropic, or Gemini keys for the LLM-judge features; everything else
runs entirely offline.
## Architecture
* **Engine** - a TypeScript HTTP API (the same Trace/Monitor/Evaluate logic the hosted product
runs) compiled to a single native binary, so no Node or Bun is needed at runtime. SQLite by
default, Postgres via one connection string, and an enterprise tier that moves span storage
to ClickHouse - see [Deployment tiers](/self-host/configuration#deployment-tiers).
* **CLI** (`agentx` / `agentx-server`) - a small Go launcher: starts the engine, opens the
dashboard, forwards shutdown signals.
* **Dashboard** - the real AgentX Governance UI, not a rebuild: the same frontend the hosted
product uses, built in self-host mode and served by the engine itself.
Get running in a minute with [Installation](/self-host/installation), then tune
[Configuration](/self-host/configuration) as needed. Every feature page in the Trace, Monitor,
Evaluation, and Improve sections applies to a self-hosted instance; pages marked "Self-host
feature" are exclusive to it.
## What's included
**Observe** - Live Traces with span-tree Timeline/Graph views and trajectory metrics,
[Sessions](/trace/sessions) with conversation-level judging, per-framework tracing guides in
the UI.
**Monitor** - built-in and custom [patterns](/monitor/patterns),
[online evaluators](/monitor/online-evaluators) (per-trace and per-session),
[custom evaluators](/monitor/custom-evaluators), [topics](/monitor/topics), signal triage with
archiving, [outcomes and judge calibration](/monitor/outcomes),
[model comparison](/monitor/model-comparison).
**Evaluate** - datasets with [version history](/evaluation/version-history),
[code scorers](/evaluation/code-scorers), LLM judge scorers, runs with judge scores and
trajectory matching, [datasets curated from production](/evaluation/datasets-from-production),
[model portability](/evaluation/model-portability), and the interactive
[Playground](/evaluation/playground) with conversation simulation.
**Improve** - [Prompt Management](/improve/prompt-management) and
[Tools](/improve/tool-schemas) registries with evidence-backed suggestions and
[measured proposal validation](/improve/validating-proposals), plus
[CI gates](/integrations/self-host-ci) with recorded history.
**Operate** - [bulk NDJSON export](/self-host/backup) of every project-scoped entity, a
[Prometheus `/metrics` endpoint](/self-host/configuration#prometheus-metrics), an append-only
[audit trail](/self-host/configuration#the-audit-trail), and a documented
[HA & DR posture](/self-host/high-availability).
Features that call an LLM - judge scoring, online evaluators, semantic
patterns, suggestion drafting, Playground runs - need a provider key
(`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY`), settable via
environment or the dashboard's Platform Settings. Trace ingest, phrase/regex
patterns, and code scorers run with no keys at all.
## What stays hosted-only
Hosted's native autotune workflow (candidate config branching for AgentX-built agents) is tied
to the hosted agent-builder and is an explicit non-goal for self-host - self-host centers on
the prompt/tool registries instead (the sidebar's **Manage → Prompts / Tools & MCPs**, with the
improvement inbox in **Insights → Suggestions**), which serve the same "make the agent better"
purpose for externally-built agents.
## Source availability and licensing
The engine and CLI are [Elastic License 2.0](https://github.com/AgentX-ai/AgentX-Trace-Eval/blob/main/LICENSE):
free for any use - including commercial self-hosting - except offering the software to third
parties as a hosted or managed service, or circumventing license-key functionality. The Python
SDK stays permissively licensed (Apache-2.0), since it ships inside your application. The
dashboard is built from AgentX's private
frontend; only its **compiled bundle** is published, attached to
[the repo's releases](https://github.com/AgentX-ai/AgentX-Trace-Eval/releases) - so installs
and source builds never need access to the private repo, and dev mode fetches the bundle
automatically on first boot.
curl | bash, pip, Docker, or from source
Ports, Postgres, auth mode, provider keys
What gets recorded on every traced call
Source, issues, and the full README
# OpenTelemetry
Source: https://developers.agentx.so/trace/opentelemetry
Send traces from any OTel-instrumented app - no AgentX SDK required
Already instrumented with OpenTelemetry? Point any OTel SDK, auto-instrumentation library, or
Collector `otlphttpexporter` straight at the engine - no AgentX SDK needed:
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4700/api/v1/otel
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=agtx_local_..." # from the engine startup log
```
## Wire formats
| Protocol | Supported | Notes |
| ------------------ | --------- | ------------------------------------------------------------------------------------------- |
| OTLP/HTTP protobuf | ✅ | The default, and the only transport Python's `opentelemetry-exporter-otlp-proto-http` ships |
| OTLP/HTTP JSON | ✅ | `OTEL_EXPORTER_OTLP_PROTOCOL=http/json` - common from Node/JS exporters |
| OTLP/gRPC | ❌ | HTTP only |
## Attribute conventions
Each incoming span becomes one trace row, with `input`/`output`/`model`/token counts pulled
from whichever convention the instrumentation actually sends:
| Convention | Attributes read |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OTel GenAI semconv | `gen_ai.input.messages` / `gen_ai.output.messages`, `gen_ai.request.model`, `gen_ai.usage.*` - current **and** legacy names (the convention has renamed fields more than once) |
| OpenLLMetry (legacy) | Indexed `gen_ai.prompt.{i}.*` / `gen_ai.completion.{i}.*` |
| OpenInference (Arize) | `input.value` / `output.value` |
| MLflow Tracing | `mlflow.spanInputs` / `mlflow.spanOutputs` / `mlflow.spanType` (JSON-encoded values unwrapped) - agents on Databricks export straight in, see [Databricks](/sdk/integrations/databricks) |
The [framework label](/trace/platform-detection) resolves from the signal itself:
`gen_ai.provider.name`, then `gen_ai.system`, then the instrumentation scope name, then
`service.name`, then the literal `otel` - so OTel traffic charts on Monitor's Platforms chart
and filters in Live Traces without any extra attribute.
Monitor runs against every OTel-ingested span by default; set `AGENTX_OTEL_MONITOR=false` to
turn that off.
## First-class citizenship
Four attributes make OTel traffic part of the full loop, not just rows in Live Traces:
* **Sessions** - set `session.id`, `gen_ai.conversation.id`, or `agentx.session_id` and traces
group into conversations on the [Sessions](/trace/sessions) surface, with session judging
applying exactly like SDK traffic. Without one, spans still group by OTel trace id.
* **Prompt identity** - set `agentx.prompt_name` (and optionally `agentx.version`) and the
whole [Improve loop](/improve/prompt-management) lights up: prompt-registry evidence
gathering and version comparison treat the trace as if the SDK had tagged it.
* **Span kinds** - `openinference.span.kind`, `gen_ai.operation.name`, `mlflow.spanType`, and
`langfuse.observation.type` are read (in that order) and stored as the span's stated
[kind](/trace/span-kinds), so a span instrumented for OpenInference or the GenAI semconv
arrives in the Execution Timeline already classified - LLM, tool, retrieval, guardrail -
instead of being inferred from names.
* **Tool calls** - a child span carrying `gen_ai.tool.name` (or an MLflow `TOOL`-typed span) is
folded up into its root interaction's `tool_calls`, with `success`/`error` derived from span
status - so Tool quality, the built-in Tool-failure check, and
[Tool Schema](/improve/tool-schemas) evidence all work on OTel traffic. In-batch only: a
parent exported in an earlier OTLP batch isn't updated retroactively.
## Verify it's flowing
Send one span, then check **Observe → Live Traces** - OTel spans arrive with their full
span tree (Timeline and Graph views) whenever the instrumentation emits parent/child links.
OTel traffic is first-class downstream too: online judge scorers sample and score it, low
verdicts raise Signals, and the traces carry judge scores in Live Traces - exactly like
SDK-ingested traffic. The runnable proof is
[`monitor_ops/07_otel_ingest_scoring.py`](https://github.com/AgentX-ai/sample-scripts) in the
sample-scripts repo: pure-OTel export, then the judge's verdict and the raised Signal read back.
Popular auto-instrumentation packages (`openinference-instrumentation-langchain`,
OpenLLMetry's SDK, MLflow Tracing) all export full span trees - one env var and your existing
instrumentation becomes AgentX's data source.
# Platform Detection
Source: https://developers.agentx.so/trace/platform-detection
Tracing is platform agnostic - every trace says which agent framework produced it, whether AgentX has an integration for it or not
AgentX traces agents on **any** platform: LangChain, CrewAI, the OpenAI Agents SDK, Google ADK,
Moveworks, an in-house runner nobody else has heard of. Every trace carries a `framework` label
saying which platform produced it, and that one label powers the Live Traces framework filter,
the framework badge on every trace row, and Monitor's **Platforms** chart (root traces per
platform over time).
The label resolves in priority order - strongest wins:
## 1. Explicit: name your platform yourself
Any string is a valid platform. This is how platforms without an AgentX integration get
first-class treatment:
```python theme={null}
with client.tracer.trace("support-agent", framework="my-inhouse-runner") as span:
span.output = run_agent(query)
```
`my-inhouse-runner` now appears in the framework filter, on trace rows, and as its own series
in the Platforms chart - exactly like a built-in. Labels are folded to lowercase on ingest, so
`"LangChain"` and `"langchain"` group as one platform.
## 2. Integrations: automatic literals
Every AgentX integration stamps its platform automatically - no parameter needed:
| Integration | Label |
| ---------------------------------------------------- | --------------- |
| `AgentXCallbackHandler` (LangChain / LangGraph) | `langchain` |
| `AgentXCrewObserver` | `crewai` |
| `AgentXTracingProcessor` (OpenAI Agents SDK) | `openai-agents` |
| `patch_openai_client` | `openai` |
| `patch_anthropic_client` | `anthropic` |
| `patch_genai_client` | `google-genai` |
| `AgentXADKPlugin` (Google ADK) | `google-adk` |
| `AgentXLiteLLMLogger` | `litellm` |
| `AgentXLlamaIndexHandler` | `llamaindex` |
| `AgentXAutoGenObserver` | `autogen` |
| `MoveworksImporter` (`agentx-moveworks` sync) | `moveworks` |
| `DatabricksTraceImporter` (`agentx-databricks` sync) | `databricks` |
A patched provider client running *inside* a span you opened yourself stamps its provider
literal onto that span too (an explicit `framework=` or a framework integration's label always
wins over it).
For OpenTelemetry ingest, the label comes from the OTel signal itself:
`gen_ai.provider.name`, then `gen_ai.system`, then the instrumentation scope name, then
`service.name`, then the literal `otel`. See [OpenTelemetry](/trace/opentelemetry).
## 3. Auto-detection: plain traces label themselves
A plain `@tracer.trace(...)` with no `framework=` and no integration in play looks at which
known orchestration framework is actually **imported** in the process (imported, not merely
installed) and labels the span when exactly one is loaded:
LangChain/LangGraph, CrewAI, LlamaIndex, AutoGen, OpenAI Agents SDK, Google ADK, Semantic
Kernel, Haystack, Pydantic AI, smolagents, DSPy.
Ambiguous (several loaded) or unknown means **no label** - the trace still ingests normally
and buckets as "Other / custom" in the dashboard. The SDK never guesses: unlabeled beats
mislabeled. Raw provider SDKs (`openai`, `anthropic`, ...) are deliberately not detected this
way - they are dependencies of nearly every framework, so their presence proves nothing about
what orchestrates the agent.
## Where the label shows up
* **Live Traces**: a framework badge on every row, plus a multi-select framework filter, and a
server-side `GET /ingest/traces?framework=...` filter.
* **Monitor → Platforms chart**: root traces per platform per time bucket
(`byFramework` on each bucket of `GET /agent-monitoring/metrics`, `frameworks` window
totals, `facets.frameworks` suggestions, and a `framework=` query filter). Unlabeled and
beyond-top-N platforms chart as "Other / custom" - every trace is always accounted for.
* **Custom evaluator webhooks**: the sampled trace's `framework` rides along in the payload.
# Sessions
Source: https://developers.agentx.so/trace/sessions
Conversation-level observability: turns, the Session Baseline Judge, and session-scoped judging
A **trace** is one interaction (a root span plus its child steps); a **session** is the
conversation those interactions belong to - every trace sharing a `session_id`. Sessions catch
the failure mode single-trace monitoring can't see: every individual reply looks fine, but the
conversation as a whole goes in circles, contradicts itself, or never resolves.
Grouping is opt-in per call - pass any stable per-conversation id:
```python theme={null}
with client.tracer.trace("support-agent", session_id=conversation_id) as span:
...
```
Traces sent without one get their own auto-generated session.
## The Sessions view
**Observe → Sessions** lists each conversation with:
| Column | Meaning |
| ------------- | --------------------------------------------------------------------------------------------------------- |
| Turns / spans | Root interactions and total recorded steps |
| Errors | Failed spans anywhere in the conversation |
| Judge Score | The **lowest** verdict among enabled session evaluators that scored it, with the judge that gave it named |
Opening a session shows every turn in order; each turn's full span tree is one click away.
![PLACEHOLDER: screenshot of the session detail dialog - a list of user/agent turns in order with per-turn latency, a Judge scores section showing two evaluator verdicts with ratings, and an 'Add as test case' button in the header.]()
## Conversation-level judging
Two kinds of judge score a session as a whole, both triggered automatically when the
conversation goes quiet:
* **Session Baseline Judge** - built-in: goal progression, consistency, non-repetition. Also
runs on demand from the session's detail view. Its rubric lives in an LLM judge scorer
(tunable via **Tune judge**); pause it from the Scorers page's row toggle.
* **Session-scoped online evaluators** - your own criteria with `scope="session"`, judged on
idle and re-judged if the conversation resumes. See
[Multi-Turn Session Evaluation](/monitor/session-evaluation).
A verdict below the evaluator's alert threshold raises a signal in the normal triage queue.
Both are also callable from the SDK - run the coherence check on demand and read the
session's spans (e.g. to walk a reported drift span up to the turn it belongs to):
```python theme={null}
score = client.monitor.sessions.coherence_check(session_id) # one judge call
print(score["rating"], score["justification"], score["driftSpanId"])
spans = client.monitor.sessions.spans(session_id) # roots + children, oldest first
```
## Where sessions come from
| Source | How |
| ---------------------- | ----------------------------------------------------------------------------------------------------------- |
| SDK | `session_id=` on `tracer.trace(...)` |
| Framework integrations | Pass `session_id` to the handler/processor constructor |
| OpenTelemetry | `session.id`, `gen_ai.conversation.id`, or `agentx.session_id` span attributes |
| Importers | `agentx-moveworks` (one session per conversation) and `agentx-databricks` (`mlflow.trace.session` metadata) |
| Playground | [Conversation simulation](/evaluation/simulate-conversation) records each run as a real `sim-` session |
## From finding to regression test
A failed conversation is a future regression test: **Add as test case** in the session detail
turns the whole conversation into a multi-turn golden dataset case - see
[Datasets from Production](/evaluation/datasets-from-production).
# Span Kinds
Source: https://developers.agentx.so/trace/span-kinds
Tell the engine what each step is - an LLM call, a tool call, a retrieval - instead of letting it guess
A trace is a tree of steps, and every surface that renders or scores it needs to know what kind
of step each span is: the Execution Timeline colours and filters by it, Code scorers branch on
it, the dashboard's span buckets count by it, and the RAG judges pull their `{context}` from it.
Spans carry that answer as a **stated kind**. Whoever produces the span says what it is, the
engine resolves it once at ingest, and every reader gets the same answer back on the wire as
`spanKind`. This is the same design as LangSmith's `run_type`, Langfuse's observation type, and
OpenInference's `openinference.span.kind`.
## The vocabulary
| Kind | What it is |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent` | An agent turn: the step that owns the whole interaction's input and output |
| `llm` | One call to a language model |
| `tool` | One tool or function execution |
| `retrieval` | A data-retrieval step: vector store, knowledge base, database lookup |
| `chain` | Glue between steps: formatting a prompt, routing, parsing |
| `embedding` | An embeddings call |
| `reranker` | Reordering retrieved candidates |
| `guardrail` | A safety or policy check |
| `evaluator` | A step that grades another step's output |
| `prompt` | Prompt construction from a template |
| `memory` | A long-term-memory operation: recalling or storing user/agent state (Mem0, Zep, Letta, or hand-rolled). Reads and writes share the kind - the span's name/metadata says which |
Kinds are mostly labels - with two that carry real semantics:
* **`retrieval` has a behavioural consequence.** The output of every retrieval span in an
interaction is what the [RAG judges](/evaluation/rag) grade against as `{context}` - a
Faithfulness or Context Relevancy scorer reads exactly those chunks.
* **`memory` is deliberately not `retrieval`**, even though a memory read looks like a lookup.
`{context}` means *knowledge the answer should be grounded in*; a recalled user preference is
*state*, not grounding - feeding it to a groundedness judge would penalize answers for not
citing it. Keeping the kinds apart lets the timeline and scorers treat "consulted the
knowledge base" and "remembered the user" as the different acts they are.
* **The bare kind string `recall` folds onto `memory`** (exact full-string match only - a tool *named* `recall_orders` is unaffected). A producer using `recall` for a knowledge-base retriever should state `retrieval` explicitly.
## How a span gets its kind
Three paths, checked in this order - an earlier one always wins.
**1. Stated explicitly.** Both the root span and child spans take `span_kind`:
```python theme={null}
with client.tracer.trace("rag-agent", span_kind="agent", sync=True) as span:
span.child_span("jailbreak_check", span_kind="guardrail",
input=query, output={"flagged": False})
span.child_span("rerank_chunks", span_kind="reranker",
input={"candidates": 8}, output={"kept": 3})
```
**2. Stamped by the SDK.** The helpers that already know what they are say so - you never pass
`span_kind` to these:
```python theme={null}
client.tracer.record_retrieval("kb_search", query=q, output=chunks) # -> retrieval
with client.tracer.trace_memory("user prefs", operation="read", query=user_id) as m:
m.output = memory.search(user_id, question) # -> memory
client.tracer.record_tool_call("charge_card", input=..., output=...) # -> tool
```
Framework integrations do the same: merged LLM steps arrive as `llm`, LangChain graph nodes as
`chain`, tool executions as `tool`, retriever runs as `retrieval`.
**3. Inferred by the engine, as a fallback.** A span that never stated a kind - a trace
recorded before span kinds existed, or a producer that just doesn't say - still classifies,
by the engine's inference ladder: a span with a `model` is `llm`, one carrying tool calls is
`tool`, the SDK's auto-generated names (`LLM Call N`, `Retrieval N`, `Memory ...`) are
recognised, and everything else is `chain`. Old traces classify exactly as they always did;
inference is a best guess, never a fact.
A **stated kind always beats the ladder**. A guardrail implemented as an LLM
call carries a model, which the ladder would read as `llm` - stating
`span_kind="guardrail"` is what keeps it a guardrail. One thing the engine
will not infer: that a root span is an `agent` turn. A flat trace's root
**is** the LLM call, so root-ness proves nothing - a root that really is an
agent turn should say so.
## Example: memory steps in a trace
One trace whose steps mix the three step families - a memory read (recalled user state), a
knowledge retrieval, and a memory write:
```python theme={null}
from agentx import AgentX
client = AgentX(api_key="...", base_url="http://localhost:4700/api/v1")
tracer = client.tracer
USER_ID = "demo-user-7"
with tracer.trace(
"travel-concierge",
input={"query": "Book my usual kind of flight to Denver next month."},
sync=True,
) as span:
# Memory READ - recalled user state, deliberately NOT a retrieval.
with tracer.trace_memory("user prefs", operation="read", query=USER_ID) as m:
m.output = ["prefers window seats", "vegetarian meals", "flies out of Oakland"]
# A knowledge lookup for contrast - this one IS a retrieval, in its own lane.
with tracer.trace_retrieval("route_search", query="Oakland to Denver flights") as r:
r.doc_count = 2
r.output = ["OAK->DEN nonstop daily 7:40", "OAK->DEN nonstop daily 18:05"]
# Memory WRITE - the agent learned something new this turn and stored it.
with tracer.trace_memory("user prefs", operation="write", query=USER_ID) as m:
m.output = "stored: planning a Denver trip next month"
span.output = "Booked the 7:40 nonstop - window seat, vegetarian meal, as usual."
```
See it from the dashboard:
![PLACEHOLDER: screenshot of the trace Execution Timeline for the travel-concierge trace - a 'user prefs' Memory step (read), a 'route_search' Retrieval step, and a second 'user prefs' Memory step (write), each with its own colored dot, and the Memory chip active in the kind filter row.]()
## Other products' vocabularies just work
The engine folds every convention it knows onto its own vocabulary, so a span instrumented for
another product classifies on arrival with no change by the producer:
| You send | Stored as | Convention |
| ----------------------------- | ----------- | ------------------ |
| `retriever` | `retrieval` | OpenInference |
| `generation` | `llm` | Langfuse |
| `chat`, `text_completion` | `llm` | OTel GenAI semconv |
| `execute_tool` | `tool` | OTel GenAI semconv |
| `invoke_agent` | `agent` | OTel GenAI semconv |
| `embeddings` | `embedding` | OTel GenAI semconv |
| `TOOL`, `AGENT`, `CHAIN`, ... | lowercased | MLflow `spanType` |
On the [OpenTelemetry ingest path](/trace/opentelemetry) the kind is read from the span's own
attributes, in this order: `openinference.span.kind`, `gen_ai.operation.name`,
`mlflow.spanType`, `langfuse.observation.type`, then `gen_ai.tool.name` (a span carrying one IS
the tool call) and finally the model attribute. Anything already instrumented with
OpenInference or the GenAI semconv arrives fully classified.
A word the engine does not recognise is **ignored, not stored**: the span falls back to the
inference ladder rather than carrying a "kind" nobody defined.
## Kind vs. `tool_calls`
These answer different questions. `span_kind="tool"` says *this span is one tool execution*.
The root trace's flat `tool_calls[]` list says *this interaction made these calls* - it is what
the built-in Tool failure check, trajectory matching, and the Tool quality column read. The SDK
maintains both for you (`trace_tool_call` writes the child span with its kind **and** appends to
the root's list); if you build spans by hand, keep doing both.