Scorer Groups

​

How the score is computed

• Renormalization: weights average over the members that actually produced a score - a deleted scorer or a judge outage shrinks the denominator instead of dragging the score down (the failed member is still reported on the result, never hidden). A group whose every member has weight 0 produces no score at all; gates still apply.
• Must-pass gates: a member marked must pass zeroes the whole group score when its goodness lands below 0.5 (a matched failure pattern, a judge under 5/10, a code score under 0.5) - “every quality bar must pass”, regardless of how good the blend looks.
• Fail-closed gates: a must-pass member that could not score at all (judge outage, deleted reference, a scorer that errors) produces no score rather than a blend that quietly reports “passed” for a safety check that never ran - the result reads No score: must-pass member "..." could not score (...).

​

Grading a dataset run with a group

group = client.monitor.scorer_groups.create(
    "Release quality bar",
    members=[
        {"kind": "judge",   "refId": quality.id, "weight": 1, "gate": False},
        {"kind": "judge",   "refId": safety.id,  "weight": 1, "gate": True},   # must pass
        {"kind": "pattern", "refId": pii.id,     "weight": 0, "gate": True},   # gate only
    ],
)

run = client.evaluations.run(
    dataset_id=dataset.id,
    subject={"kind": "custom_agent", "displayName": "my-agent"},
    scorer_group_id=group.id,
).execute(my_agent).finalize()

run.gate(fail_under=7)                    # gates on the GROUP score, like any rating
run.gate(fail_under=8, scorer="Safety")   # or gate a named JUDGE member's own average

​

Scoring live traffic

​

Scoring each trace

client.monitor.scorer_groups.update(group.id, online={
    "enabled": True,
    "sampleRate": 0.2,        # 20% of traffic
    "alertThreshold": 6,      # Signal when the GROUP score lands below 6/10
    "severity": "high",       # omitted keys keep their stored value - send "scope": "trace" to flip a session group back
})

client.monitor.scorer_groups.ratings(group.id, window="7d")  # score history for the chart

• Score chip: when a group has scored a trace, the group score owns the Score chip in Live Traces (the composed final opinion beats any individual verdict for the headline); individual verdicts stay in the expanded row.
• Trace Details: when both kinds scored the trace, Trace Details shows a Group score / Judge scores switch - the group aggregate with its blend breakdown on one side, per-member verdicts (recorded individually at scoring time) on the other. With only one kind present, the section shows it directly, no toggle.
• Webhooks: a below-threshold group score pages the agent’s alert channels exactly like a matched pattern or a low online-evaluator verdict.
• Judge budget: judge members draw from the same online judge-call budget online evaluators reserve from - a group with 3 judge members takes 3 slots per sampled trace, and an exhausted budget skips the group rather than scoring it on a partial panel.
• KPIs: group scores (and member verdicts) are ratings, not runs - they never inflate the Overview’s run counts.

​

Scoring whole sessions

client.monitor.scorer_groups.update(group.id, online={
    "enabled": True,
    "sampleRate": 1.0,        # of eligible idle sessions
    "scope": "session",
    "idleSeconds": 120,       # judge once quiet for 2 minutes
    "alertThreshold": 6,      # Signal when the conversation's GROUP score lands below 6/10
    "severity": "high",
})

• Judge members are asked the structured whole-session question built from their criteria (consistency across turns, whether the user’s need was resolved). A judge’s per-trace judgePrompt never applies at session scope - the same rule session evaluators follow.
• Pattern and code/external members get the conversation split by speaker, not the formatted transcript: every user turn joined together as their input, every agent turn joined together as their output - so a “response contains” condition reads everything the agent said across the conversation without ever firing on the user’s own words. Both sides are read from root spans first, with fallbacks for OTel-style sessions whose roots are empty: the user side falls back to the full formatted transcript (over-matching is the honest direction for a tripwire), while the agent side falls back only to child span outputs - genuinely the agent’s words - and stays empty when even those are missing. Tool calls are not passed, and a code scorer’s spans array covers only the session’s last (anchor) trace, not every turn.
• Weights, renormalization, and must-pass gates work identically - an apology tripwire pattern with weight: 0, gate: true zeroes a conversation that apologizes anywhere.

client.monitor.sessions.scores(session_id)
# -> [{"kind": "scorer-group:<groupId>", "rating": 4.5, "justification": "Weighted blend: ..."}, ...]

​

SDK surface