Code Scorers

Evaluate’s judge scoring and similarity metrics (vector/Jaccard/BLEU/ROUGE) cover most grading needs, but sometimes what you want to check is exact code, not a model’s opinion - did the output actually call this function, is this valid JSON, does the response stay under N words. Datasets and LLM judge scorers (their offline eval config) both support attaching one or more code scorers: plain JavaScript functions, run in-process (node:vm, sandboxed - no network, no filesystem, a 3-second timeout) against each result’s input/output/expected - plus toolCalls (the linked trace’s recorded tool calls) when the result carries a trace_id, so a scorer can assert on tool behavior. They run after the judge and similarity metrics, so a scorer also receives scores - the result’s other verdicts - and can combine them (see below). Live traffic has a counterpart - the code scorer kind on the Scorers page - but the shapes differ: an offline code scorer is a synchronous JavaScript function body like the ones below, while a live code scorer defines an async handler() and may also be Python. From the dashboard: attach them in the dataset editor (opened from the sidebar’s Manage → Datasets) or in a judge scorer’s Offline runs step. Each one gets a name and a JavaScript function body. From the SDK (dataset builder, so the scorer is versioned with the dataset it guards):

dataset = client.evaluations.datasets.builder(
    name="Support gate",
    code_scorers=[{
        "name": "mentions no competitor",
        "code": "const bad = /rivalcorp/i.test(output); return { score: bad ? 0 : 1 };",
    }],
).add_case(query="...", expected_results="...").publish()

const wordCount = output.trim().split(/\s+/).length;
if (wordCount <= 60) {
  return { score: 1, reasoning: `Concise: ${wordCount} words.` };
}
return { score: 0.5, reasoning: `Wordy: ${wordCount} words (over the 60-word guideline).` };

Return a bare number (0-1) or { score, reasoning }. A scorer that throws, times out, or returns something unexpected degrades to { score: null, error } for that one result - it never blocks the judge rating or any other score from being computed.

Combining judges and metrics into one final score

Because code scorers run after the judges and metrics, each one receives a scores object with the result’s other verdicts:

scores = {
  rating: number | null,                    // primary judge, 0-10
  judges: { [scorerName]: number | null },  // additional judge scorers, 0-10 each
  vectorSimilarity: number | null,          // cosine, 0-1 (null when not enabled)
  jaccardSimilarity: number | null,
  bleuScore: number | null,
  rougeScore: number | null,
}

Direction check before weighting: all four built-in metrics are higher-is-better (1 = identical to expected, 0 = nothing in common), and judge ratings are usually higher-is-better too - but a judge whose rubric measures a bad quality (a “Toxicity” judge where 10 = worst) must be inverted before it joins the blend (1 - value / 10). The dashboard template has a higherIsBetter flag per entry for exactly this. That makes a custom weighted final score a few lines - your own blend of judge verdicts and deterministic metrics:

// 50% primary judge, 20% cosine, 30% ROUGE - renormalized over whatever is available.
const WEIGHTS = { rating: 0.5, vectorSimilarity: 0.2, rougeScore: 0.3 };
if (!scores) return { score: null };
let total = 0, used = 0;
if (scores.rating != null) { total += (scores.rating / 10) * WEIGHTS.rating; used += WEIGHTS.rating; }
for (const m of ["vectorSimilarity", "rougeScore"]) {
  if (scores[m] != null) { total += scores[m] * WEIGHTS[m]; used += WEIGHTS[m]; }
}
return used > 0 ? { score: total / used } : { score: null };

The dashboard’s code scorer editor ships this as the Weighted final score template (plus Worst judge wins, which takes the lowest verdict across the primary and every additional scorer). On a multi-judge run, scores.judges carries each additional scorer’s rating by name, so per-judge weights work too. scores is undefined on surfaces that don’t score judges first - always null-check it. From the SDK, attach scorers when building a grading config:

settings = client.evaluations.settings.builder(
    name="Strict grading",
    acceptance_criteria="...",
    code_scorers=[
        {
            "name": "conciseness",
            "enabled": True,
            "code": (
                "const wordCount = output.trim().split(/\\s+/).length;\n"
                "return wordCount <= 60"
                "  ? { score: 1, reasoning: `Concise: ${wordCount} words.` }"
                "  : { score: 0.5, reasoning: `Wordy: ${wordCount} words.` };"
            ),
        }
    ],
).publish()

Reading them back

Code scorers are fully retrievable through the SDK - both the stored configuration and the per-result outputs:

# The stored configuration, from either home it can live in:
scorer = client.monitor.judge_scorers.get(scorer_id)
scorer.code_scorers        # [{ "name": "Final score", "code": "...", "enabled": True }, ...]

dataset = client.evaluations.datasets.get(dataset_id)
dataset.code_scorers       # same shape; None when the dataset has none

# The per-result outputs, one row per enabled scorer:
run = client.evaluations.run(...).execute(my_agent).finalize()
for row in run.results():
    row.code_scorer_results  # [{ "name", "score", "reasoning"?, "error"? }, ...]

datasets.get() also round-trips them through import_dataset(), so copying a dataset carries its scorers along. Remember which home applies at run time: a run graded by an LLM judge scorer uses that scorer’s code scorers; only a run with no scorer selected falls back to the dataset’s own. Each result’s scorer rows come back via run.results() as codeScorerResults. The runnable versions are sample-scripts/selfhost_demo/03_evaluate_with_a_dataset.py and sample-scripts/eval_deep_dive/08_weighted_final_score.py.

Expected-trajectory matching reports through the same per-result scorer rows: a case with expected_tools shows a “Trajectory match (mode)” row next to your own code scorers, pass/fail with the expected-vs-actual call lists in its reasoning.

The Scorer Catalog

Judge Calibration & Tuning

Combining judges and metrics into one final score

Reading them back

​Combining judges and metrics into one final score

​Reading them back

Combining judges and metrics into one final score

Reading them back