> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.
# External & Code Scorers
> Your own scoring logic - as an HTTP endpoint, or a script run in-engine
For checks that need your own logic - a proprietary classifier, a business rule that needs data
outside the trace, anything code can decide that a phrase/regex/semantic pattern can't - two
scorer kinds run YOUR code against a sample of live traffic:
* An **external scorer** POSTs each sampled trace to your own HTTP endpoint and uses its verdict.
* A **code scorer** runs a script you write (Python or JavaScript) inside the engine itself - no
endpoint to host.
Like [LLM judge scorers](/monitor/online-evaluators), both are standalone entities on the
[Scorers page](/monitor/scorers) with their own sample rate, agent scope, enable/pause toggle,
and severity. (The SDK/API resource name for both remains `custom-evaluators`.)
## Code scorers
**Scorers → New scorer → Code scorer.** The script defines one function:
```python theme={null}
from typing import Any
# handler returns a numeric score between 0 and 1,
# or a dict with 'score' and optional 'metadata' and 'name' fields,
# or None to skip scoring this trace.
async def handler(
input: Any,
output: Any,
expected: Any, # always None for live traffic
metadata: dict[str, Any],
trace: Any,
) -> float | dict[str, Any] | None:
all_spans = await trace.get_spans()
llm_spans = await trace.get_spans(span_type=['llm'])
return {
'name': 'span count scorer',
'score': 1.0 if output else 0.0,
'metadata': {
'total_span_count': len(all_spans),
'llm_span_count': len(llm_spans),
},
}
```
The JavaScript variant is identical in shape (`async function handler(...)`,
`await trace.getSpans({ spanType: ["llm"] })`). A returned score **below the scorer's alert
threshold** (default 0.5) raises a signal; every scored check is recorded in the scorer's event
history either way, and `None`/`null` skips the trace entirely.
`trace.get_spans()` returns the trace's own span subtree (root first, start-time ordered). Each
span carries `span_id`, `parent_span_id`, `name`, `input`, `output`, `error`, `model`,
`latency_ms`, `input_tokens`, `output_tokens`, `tool_calls`, `metadata`, `started_at`, and a
derived `type`: `llm` (a model is recorded), `tool` (tool calls recorded), `retrieval`
(`metadata.kind == "retrieval"`), or `span` (anything else).
| Behavior | Detail |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Runtime | JavaScript: in-process sandbox. Python: a `python3` subprocess (present in the Docker image; on bare installs the scorer reports clearly if `python3` is missing) |
| Budget | 8 seconds per trace, per scorer |
| Failures | A crashing/timing-out script records an errored check in the scorer's history and never raises a signal or breaks ingest |
| Security | Not a sandbox boundary: a scorer is code the operator chose to run on their own engine. Don't hand scorer creation to anyone who shouldn't run code on the machine |
**Dry run** in the dialog executes the script against a synthetic two-span sample (one root, one
`llm` span) and shows the score it returns.
## External scorers
**Request** (engine → your endpoint, `POST`, JSON, `schemaVersion: 2`):
```json theme={null}
{
"schemaVersion": 2,
"evaluatorId": "abc123",
"evaluatorName": "Policy checker",
"agentId": "vN2k...",
"traceId": "trace-id",
"trace": {
"input": "...", "output": "...", "error": null, "toolCalls": [],
"name": "support-agent", "model": "gpt-4o-mini", "framework": "openai",
"sessionId": "sess-1", "spanId": "root-span",
"latencyMs": 1450, "inputTokens": 220, "outputTokens": 64,
"cacheReadTokens": null, "cacheWriteTokens": null,
"metadata": { "channel": "chat" },
"startedAt": "2026-08-22T01:02:03.000Z", "createdAt": "2026-08-22T01:02:04.000Z"
},
"spans": [
{ "span_id": "root-span", "parent_span_id": null, "name": "support-agent", "type": "span", "input": "...", "output": "...", "error": null, "model": null, "latency_ms": 1450, "input_tokens": null, "output_tokens": null, "tool_calls": null, "metadata": null, "started_at": "..." },
{ "span_id": "llm-1", "parent_span_id": "root-span", "name": "LLM Call 1", "type": "llm", "model": "gpt-4o-mini", "latency_ms": 900, "input_tokens": 220, "output_tokens": 64, "input": "...", "output": "...", "error": null, "tool_calls": null, "metadata": null, "started_at": "..." }
]
}
```
Everything the tracer records rides along: the full root record under `trace` (the v1 keys -
`input`/`output`/`error`/`toolCalls` - are exactly where they always were, so v1 endpoints keep
working untouched), and the trace's span subtree under `spans` with the same per-span fields and
derived `type` values code scorers see (`llm`/`tool`/`retrieval`/`span`).
Two things endpoint authors trip on: `agentId` is AgentX's **internal agent id**, never the
agent's name (fetch the agent list once rather than string-comparing names). And
`evaluatorId`/`agentId`/`traceId` are all **`null` in a dry run** - the first payload your
endpoint will ever receive - so handle nulls before assuming strings.
**Response your endpoint must return:**
```json theme={null}
{ "matches": true, "reason": "optional, shown on the resulting signal", "score": 7.5 }
```
`matches` is the only field that decides anything. `reason` and `score` are both optional and purely informational - recorded and shown alongside the resulting signal, but neither affects whether one is raised. A minimal example endpoint (Flask):
```python theme={null}
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.post("/evaluate")
def evaluate():
payload = request.json
output = payload["trace"]["output"] or ""
matched = "cannot help" in output.lower()
return jsonify({"matches": matched, "reason": "contains a refusal phrase" if matched else None})
```
## Setup and behavior
Build it from the dashboard: **Scorers → New scorer → External scorer**. The dialog
documents the request/response contract in place, and **Dry run** sends a synthetic v2 payload to
your URL so you can confirm it responds correctly before saving.
| Behavior | Detail |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Raise when | Your choice of `matches: true` or `false` raising the signal (defaults to `true`), with a severity like any evaluator |
| Timeout | Fixed 8 seconds |
| Failures | An erroring/timing-out/malformed response skips that one check for that trace (logged, not raised); every other evaluator still runs |
| Auth | None built in - embed a token in the URL's query string, the same way a Slack incoming-webhook URL does |
Distinct from `monitor_profiles.channels`' `"webhook:"` notification targets - those fire-and-forget a message *after* a signal is raised and never consume a response. A custom evaluator's response *is* the detection result, awaited synchronously (up to the 8-second timeout).
Dashboard-only for now - no SDK method exists yet. The REST API works directly if you want to script it: full CRUD on `/agent-monitoring/custom-evaluators[/:id]` (code scorers pass `{"kind": "code", "language": "python"|"javascript", "script": "...", "alertBelow": 0.5}`), plus `POST /agent-monitoring/custom-evaluators/dry-run` (transient - `{"url": ...}` for external, `{"kind": "code", "language": ..., "script": ...}` for code - the live response back out, nothing persisted).