External & Code Scorers

For checks that need your own logic - a proprietary classifier, a business rule that needs data outside the trace, anything code can decide that a phrase/regex/semantic pattern can’t - two scorer kinds run YOUR code against a sample of live traffic:

An external scorer POSTs each sampled trace to your own HTTP endpoint and uses its verdict.
A code scorer runs a script you write (Python or JavaScript) inside the engine itself - no endpoint to host.

Like LLM judge scorers, both are standalone entities on the Scorers page with their own sample rate, agent scope, enable/pause toggle, and severity. (The SDK/API resource name for both remains custom-evaluators.)

Code scorers

Scorers → New scorer → Code scorer. The script defines one function:

from typing import Any

# handler returns a numeric score between 0 and 1,
# or a dict with 'score' and optional 'metadata' and 'name' fields,
# or None to skip scoring this trace.
async def handler(
    input: Any,
    output: Any,
    expected: Any,       # always None for live traffic
    metadata: dict[str, Any],
    trace: Any,
) -> float | dict[str, Any] | None:
    all_spans = await trace.get_spans()
    llm_spans = await trace.get_spans(span_type=['llm'])
    return {
        'name': 'span count scorer',
        'score': 1.0 if output else 0.0,
        'metadata': {
            'total_span_count': len(all_spans),
            'llm_span_count': len(llm_spans),
        },
    }

The JavaScript variant is identical in shape (async function handler(...), await trace.getSpans({ spanType: ["llm"] })). A returned score below the scorer’s alert threshold (default 0.5) raises a signal; every scored check is recorded in the scorer’s event history either way, and None/null skips the trace entirely. trace.get_spans() returns the trace’s own span subtree (root first, start-time ordered). Each span carries span_id, parent_span_id, name, input, output, error, model, latency_ms, input_tokens, output_tokens, tool_calls, metadata, started_at, and a derived type - the engine-wide span kind (agent, llm, tool, retrieval, chain, embedding, reranker, guardrail, evaluator, prompt, memory). Every span, the root included, classifies through the same ladder: a producer-stated span_kind always wins, a span with a model infers llm, one with tool calls infers tool, and an unclassifiable span is chain; the older "span" value is retired and never emitted, the Dry run synthetic sample included (its root is "agent"). Match the root by its parent_span_id being null, not by type.

Behavior	Detail
Runtime	JavaScript: in-process sandbox. Python: a `python3` subprocess (present in the Docker image; on bare installs the scorer reports clearly if `python3` is missing)
Budget	8 seconds per trace, per scorer
Failures	A crashing/timing-out script records an errored check in the scorer’s history and never raises a signal or breaks ingest
Security	Not a sandbox boundary: a scorer is code the operator chose to run on their own engine. Don’t hand scorer creation to anyone who shouldn’t run code on the machine

Dry run in the dialog executes the script against a synthetic two-span sample (one root, one llm span) and shows the score it returns.

External scorers

Request (engine → your endpoint, POST, JSON, schemaVersion: 2):

{
  "schemaVersion": 2,
  "evaluatorId": "abc123",
  "evaluatorName": "Policy checker",
  "agentId": "vN2k...",
  "traceId": "trace-id",
  "trace": {
    "input": "...", "output": "...", "error": null, "toolCalls": [],
    "name": "support-agent", "model": "gpt-4o-mini", "framework": "openai",
    "sessionId": "sess-1", "spanId": "root-span",
    "latencyMs": 1450, "inputTokens": 220, "outputTokens": 64,
    "cacheReadTokens": null, "cacheWriteTokens": null,
    "metadata": { "channel": "chat" },
    "startedAt": "2026-08-22T01:02:03.000Z", "createdAt": "2026-08-22T01:02:04.000Z"
  },
  "spans": [
    { "span_id": "root-span", "parent_span_id": null, "name": "support-agent", "type": "chain", "input": "...", "output": "...", "error": null, "model": null, "latency_ms": 1450, "input_tokens": null, "output_tokens": null, "tool_calls": null, "metadata": null, "started_at": "..." },
    { "span_id": "llm-1", "parent_span_id": "root-span", "name": "LLM Call 1", "type": "llm", "model": "gpt-4o-mini", "latency_ms": 900, "input_tokens": 220, "output_tokens": 64, "input": "...", "output": "...", "error": null, "tool_calls": null, "metadata": null, "started_at": "..." }
  ]
}

Everything the tracer records rides along: the full root record under trace (the v1 keys - input/output/error/toolCalls - are exactly where they always were, so v1 endpoints keep working untouched), and the trace’s span subtree under spans with the same per-span fields and derived type values code scorers see (the full span-kind vocabulary - llm, tool, retrieval, chain, memory, …).

Two things endpoint authors trip on: agentId is AgentX’s internal agent id, never the agent’s name (fetch the agent list once rather than string-comparing names). And evaluatorId/agentId/traceId are all null in a dry run - the first payload your endpoint will ever receive - so handle nulls before assuming strings.

Response your endpoint must return:

{ "matches": true, "reason": "optional, shown on the resulting signal", "score": 7.5 }

matches is the only field that decides anything. reason and score are both optional and purely informational - recorded and shown alongside the resulting signal, but neither affects whether one is raised. A minimal example endpoint (Flask):

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.post("/evaluate")
def evaluate():
    payload = request.json
    output = payload["trace"]["output"] or ""
    matched = "cannot help" in output.lower()
    return jsonify({"matches": matched, "reason": "contains a refusal phrase" if matched else None})

Setup and behavior

Build it from the dashboard: Scorers → New scorer → External scorer. The dialog documents the request/response contract in place, and Dry run sends a synthetic v2 payload to your URL so you can confirm it responds correctly before saving.

Behavior	Detail
Raise when	Your choice of `matches: true` or `false` raising the signal (defaults to `true`), with a severity like any evaluator
Timeout	Fixed 8 seconds
Failures	An erroring/timing-out/malformed response skips that one check for that trace (logged, not raised); every other evaluator still runs
Auth	None built in - embed a token in the URL’s query string, the same way a Slack incoming-webhook URL does

External scorer URLs pass the engine’s egress guard: http(s) only, cloud metadata endpoints are always refused, and private/loopback targets are blocked on multi-tenant deployments (on regular self-host they stay allowed - pointing a scorer at your own local endpoint is the normal deployment shape). A blocked URL is a 400 at save time, not a silent failure at check time.

Distinct from monitor_profiles.channels’ "webhook:<url>" notification targets - those fire-and-forget a message after a signal is raised and never consume a response. A custom evaluator’s response is the detection result, awaited synchronously (up to the 8-second timeout).

The Python SDK covers the whole lifecycle as client.monitor.scorers:

client.monitor.scorers.list()
client.monitor.scorers.create_code("Apology overload", script, language="python",
                                   sample_rate=0.5, alert_below=0.5)
client.monitor.scorers.create_external("Policy endpoint", "https://scorer.internal/score")
client.monitor.scorers.update(scorer_id, enabled=False)
client.monitor.scorers.events(scorer_id, window="24h")     # per-check history
client.monitor.scorers.dry_run(kind="code", language="python", script="...")  # nothing persisted
client.monitor.scorers.delete(scorer_id)

# The shipped template scorers live on the same surface:
client.monitor.scorers.templates()                         # keys, rules, enabled state
client.monitor.scorers.enable(["pii-in-response"])
client.monitor.scorers.disable(["profanity-in-response"])

sample_rate defaults to 0.1 for both kinds - every check is a real outbound HTTP call or a script execution, so sampling starts conservative.
The REST equivalents live on /agent-monitoring/custom-evaluators: GET/POST on the collection, PUT/DELETE on /:id, and GET /:id/events for the check history (code scorers pass {"kind": "code", "language": "python"|"javascript", "script": "...", "alertBelow": 0.5}).
POST /agent-monitoring/custom-evaluators/dry-run is transient - {"url": ...} for external, {"kind": "code", "language": ..., "script": ...} for code - the live response back out, nothing persisted.

The Scorer Catalog

Judge Calibration & Tuning

Live Traffic Scoring

External & Code Scorers

Code scorers

External scorers

Setup and behavior

​Code scorers

​External scorers

​Setup and behavior

Code scorers

External scorers

Setup and behavior