Code Scorers

Evaluate’s judge scoring and similarity metrics (vector/Jaccard/BLEU/ROUGE) cover most grading needs, but sometimes what you want to check is exact code, not a model’s opinion - did the output actually call this function, is this valid JSON, does the response stay under N words. Datasets and Evaluator configs both support attaching one or more code scorers: plain JavaScript functions, run in-process (node:vm, sandboxed - no network, no filesystem, a 3-second timeout) against each result’s input/output/expected - plus toolCalls (the linked trace’s recorded tool calls) when the result carries a trace_id, so a scorer can assert on tool behavior. They run right alongside the judge and similarity metrics, not instead of them. From the dashboard: Governance → Evaluate → Datasets (or Evaluator) → create or edit → Code Scorers section → Add scorer. Each one gets a name and a JavaScript function body: From the SDK (dataset builder, so the scorer is versioned with the dataset it guards):

dataset = client.evaluations.datasets.builder(
    name="Support gate",
    code_scorers=[{
        "name": "mentions no competitor",
        "code": "const bad = /rivalcorp/i.test(output); return { score: bad ? 0 : 1 };",
    }],
).add_case(query="...", expected_results="...").publish()

const wordCount = output.trim().split(/\s+/).length;
if (wordCount <= 60) {
  return { score: 1, reasoning: `Concise: ${wordCount} words.` };
}
return { score: 0.5, reasoning: `Wordy: ${wordCount} words (over the 60-word guideline).` };

Return a bare number (0-1) or { score, reasoning }. A scorer that throws, times out, or returns something unexpected degrades to { score: null, error } for that one result - it never blocks the judge rating or any other score from being computed. From the SDK, attach scorers when building a grading config:

settings = client.evaluations.settings.builder(
    name="Strict grading",
    acceptance_criteria="...",
    code_scorers=[
        {
            "name": "conciseness",
            "enabled": True,
            "code": (
                "const wordCount = output.trim().split(/\\s+/).length;\n"
                "return wordCount <= 60"
                "  ? { score: 1, reasoning: `Concise: ${wordCount} words.` }"
                "  : { score: 0.5, reasoning: `Wordy: ${wordCount} words.` };"
            ),
        }
    ],
).publish()

Each result’s scorer rows come back via run.results() as codeScorerResults. The runnable version is sample-scripts/selfhost_demo/03_evaluate_with_a_dataset.py.

Expected-trajectory matching reports through the same per-result scorer rows: a case with expected_tools shows a “Trajectory match (mode)” row next to your own code scorers, pass/fail with the expected-vs-actual call lists in its reasoning.

Pre-Deployment Quality Gate

Build & Run

Scoring & Datasets

Framework Examples