External & Code Scorers

For checks that need your own logic - a proprietary classifier, a business rule that needs data outside the trace, anything code can decide that a phrase/regex/semantic pattern can’t - two scorer kinds run YOUR code against a sample of live traffic:

An external scorer POSTs each sampled trace to your own HTTP endpoint and uses its verdict.
A code scorer runs a script you write (Python or JavaScript) inside the engine itself - no endpoint to host.

Like LLM judge scorers, both are standalone entities on the Scorers page with their own sample rate, agent scope, enable/pause toggle, and severity. (The SDK/API resource name for both remains custom-evaluators.)

Code scorers

Scorers → New scorer → Code scorer. The script defines one function:

from typing import Any

# handler returns a numeric score between 0 and 1,
# or a dict with 'score' and optional 'metadata' and 'name' fields,
# or None to skip scoring this trace.
async def handler(
    input: Any,
    output: Any,
    expected: Any,       # always None for live traffic
    metadata: dict[str, Any],
    trace: Any,
) -> float | dict[str, Any] | None:
    all_spans = await trace.get_spans()
    llm_spans = await trace.get_spans(span_type=['llm'])
    return {
        'name': 'span count scorer',
        'score': 1.0 if output else 0.0,
        'metadata': {
            'total_span_count': len(all_spans),
            'llm_span_count': len(llm_spans),
        },
    }

The JavaScript variant is identical in shape (async function handler(...), await trace.getSpans({ spanType: ["llm"] })). A returned score below the scorer’s alert threshold (default 0.5) raises a signal; every scored check is recorded in the scorer’s event history either way, and None/null skips the trace entirely. trace.get_spans() returns the trace’s own span subtree (root first, start-time ordered). Each span carries span_id, parent_span_id, name, input, output, error, model, latency_ms, input_tokens, output_tokens, tool_calls, metadata, started_at, and a derived type: llm (a model is recorded), tool (tool calls recorded), retrieval (metadata.kind == "retrieval"), or span (anything else).

Behavior	Detail
Runtime	JavaScript: in-process sandbox. Python: a `python3` subprocess (present in the Docker image; on bare installs the scorer reports clearly if `python3` is missing)
Budget	8 seconds per trace, per scorer
Failures	A crashing/timing-out script records an errored check in the scorer’s history and never raises a signal or breaks ingest
Security	Not a sandbox boundary: a scorer is code the operator chose to run on their own engine. Don’t hand scorer creation to anyone who shouldn’t run code on the machine

Dry run in the dialog executes the script against a synthetic two-span sample (one root, one llm span) and shows the score it returns.

External scorers

Request (engine → your endpoint, POST, JSON, schemaVersion: 2):

{
  "schemaVersion": 2,
  "evaluatorId": "abc123",
  "evaluatorName": "Policy checker",
  "agentId": "vN2k...",
  "traceId": "trace-id",
  "trace": {
    "input": "...", "output": "...", "error": null, "toolCalls": [],
    "name": "support-agent", "model": "gpt-4o-mini", "framework": "openai",
    "sessionId": "sess-1", "spanId": "root-span",
    "latencyMs": 1450, "inputTokens": 220, "outputTokens": 64,
    "cacheReadTokens": null, "cacheWriteTokens": null,
    "metadata": { "channel": "chat" },
    "startedAt": "2026-08-22T01:02:03.000Z", "createdAt": "2026-08-22T01:02:04.000Z"
  },
  "spans": [
    { "span_id": "root-span", "parent_span_id": null, "name": "support-agent", "type": "span", "input": "...", "output": "...", "error": null, "model": null, "latency_ms": 1450, "input_tokens": null, "output_tokens": null, "tool_calls": null, "metadata": null, "started_at": "..." },
    { "span_id": "llm-1", "parent_span_id": "root-span", "name": "LLM Call 1", "type": "llm", "model": "gpt-4o-mini", "latency_ms": 900, "input_tokens": 220, "output_tokens": 64, "input": "...", "output": "...", "error": null, "tool_calls": null, "metadata": null, "started_at": "..." }
  ]
}

Everything the tracer records rides along: the full root record under trace (the v1 keys - input/output/error/toolCalls - are exactly where they always were, so v1 endpoints keep working untouched), and the trace’s span subtree under spans with the same per-span fields and derived type values code scorers see (llm/tool/retrieval/span).

Two things endpoint authors trip on: agentId is AgentX’s internal agent id, never the agent’s name (fetch the agent list once rather than string-comparing names). And evaluatorId/agentId/traceId are all null in a dry run - the first payload your endpoint will ever receive - so handle nulls before assuming strings.

Response your endpoint must return:

{ "matches": true, "reason": "optional, shown on the resulting signal", "score": 7.5 }

matches is the only field that decides anything. reason and score are both optional and purely informational - recorded and shown alongside the resulting signal, but neither affects whether one is raised. A minimal example endpoint (Flask):

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.post("/evaluate")
def evaluate():
    payload = request.json
    output = payload["trace"]["output"] or ""
    matched = "cannot help" in output.lower()
    return jsonify({"matches": matched, "reason": "contains a refusal phrase" if matched else None})

Setup and behavior

Build it from the dashboard: Scorers → New scorer → External scorer. The dialog documents the request/response contract in place, and Dry run sends a synthetic v2 payload to your URL so you can confirm it responds correctly before saving.

Behavior	Detail
Raise when	Your choice of `matches: true` or `false` raising the signal (defaults to `true`), with a severity like any evaluator
Timeout	Fixed 8 seconds
Failures	An erroring/timing-out/malformed response skips that one check for that trace (logged, not raised); every other evaluator still runs
Auth	None built in - embed a token in the URL’s query string, the same way a Slack incoming-webhook URL does

Distinct from monitor_profiles.channels’ "webhook:<url>" notification targets - those fire-and-forget a message after a signal is raised and never consume a response. A custom evaluator’s response is the detection result, awaited synchronously (up to the 8-second timeout).

Dashboard-only for now - no SDK method exists yet. The REST API works directly if you want to script it: full CRUD on /agent-monitoring/custom-evaluators[/:id] (code scorers pass {"kind": "code", "language": "python"|"javascript", "script": "...", "alertBelow": 0.5}), plus POST /agent-monitoring/custom-evaluators/dry-run (transient - {"url": ...} for external, {"kind": "code", "language": ..., "script": ...} for code - the live response back out, nothing persisted).

Live Traffic Scoring

Ground Truth & Calibration

External & Code Scorers

Code scorers

External scorers

Setup and behavior

​Code scorers

​External scorers

​Setup and behavior

Code scorers

External scorers

Setup and behavior