> ## Documentation Index > Fetch the complete documentation index at: https://developers.agentx.so/llms.txt > Use this file to discover all available pages before exploring further. # Scorer Groups > Compose scorers of any kind into one 0-10 score with weights and must-pass gates A **scorer group** composes several scorers - LLM judges, patterns, code and external scorers - into **one 0-10 score**. Members are stored by reference, so a group reuses the scorers your project already has: editing a member scorer changes every group that includes it. The group itself is then what a dataset run or live traffic is graded with, which answers the question per-scorer configuration never could: *"I have five opinions about this response - what is THE score?"* Create and manage groups on the **Scorers** page's **Groups** sub-tab. ## How the score is computed Every member keeps its native contract and is normalized to a 0-1 "goodness" before weighting: | Member kind | Native scale | Normalized goodness | | ---------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | LLM judge | 0-10 rating | rating ÷ 10 | | Pattern (templates included) | matched / not matched | polarity-mapped: a matched **failure** pattern is 0, clean is 1; a matched **proper** pattern ("contains disclaimer") is 1 | | Code scorer | 0..1 | as-is | | External scorer | 0..1 score, or a boolean verdict | score as-is; a boolean maps 0/1 through the scorer's invert setting | On the wire a member is `{ "kind", "refId", "weight", "gate" }`, with exactly three `kind` values: `judge` (`refId` is an LLM judge scorer id), `pattern` (a pattern id), and `custom` (a custom-evaluator id - this one kind covers both code and external scorers, since either is a custom evaluator under the hood). The group score is the **weighted average of member goodness, reported on 0-10** - so it reads like a rating everywhere ratings already live: the run rating column, CI gates (`gate(fail_under=...)`), and live alert thresholds. Three safeguards: * **Renormalization**: weights average over the members that actually produced a score - a deleted scorer or a judge outage shrinks the denominator instead of dragging the score down (the failed member is still reported on the result, never hidden). A group whose every member has weight 0 produces no score at all; gates still apply. * **Must-pass gates**: a member marked *must pass* zeroes the whole group score when its goodness lands below 0.5 (a matched failure pattern, a judge under 5/10, a code score under 0.5) - "every quality bar must pass", regardless of how good the blend looks. * **Fail-closed gates**: a must-pass member that could not score at all (judge outage, deleted reference, a scorer that errors) produces **no score** rather than a blend that quietly reports "passed" for a safety check that never ran - the result reads `No score: must-pass member "..." could not score (...)`. A judge member contributes its **rubric verdict only**: its similarity metrics and [attached code checks](/evaluation/code-scorers) belong to that scorer's own grading recipe and do not run inside a group. Blending metrics into a score stays an attached-code-check job, scoped to the scorer that owns those metrics. ## Grading a dataset run with a group Pass the group's id as `scorer_group_id` (instead of `scorer_id`) - the group's aggregate fills the rating column, judge members surface as labeled per-row verdicts, and deterministic members as scorer rows: ```python theme={null} group = client.monitor.scorer_groups.create( "Release quality bar", members=[ {"kind": "judge", "refId": quality.id, "weight": 1, "gate": False}, {"kind": "judge", "refId": safety.id, "weight": 1, "gate": True}, # must pass {"kind": "pattern", "refId": pii.id, "weight": 0, "gate": True}, # gate only ], ) run = client.evaluations.run( dataset_id=dataset.id, subject={"kind": "custom_agent", "displayName": "my-agent"}, scorer_group_id=group.id, ).execute(my_agent).finalize() run.gate(fail_under=7) # gates on the GROUP score, like any rating run.gate(fail_under=8, scorer="Safety") # or gate a named JUDGE member's own average ``` `scorer_group_id` and `scorer_id` are mutually exclusive - the group wins when both are sent, and a group-graded run also nulls out any `additional_scorer_ids`: the group is the whole grading recipe, not a layer on top of one. Member verdicts land per row (judges in `judge_scorer_results`, deterministic members as scorer rows), the run's breakdown leads with "*Name* (group)", and the Evaluate list's Scorers column shows the group name. A **non-gating** member deleted after the group was built degrades to "not scored" on future results rather than failing the run; a deleted **must-pass** member fails the group closed instead - no score, per the fail-closed rule above. In the dashboard, the Create evaluation dialog offers your groups next to the judge scorer picker ("Or grade with a scorer group"), and the generated snippet emits `scorer_group_id=`. The **Playground**'s Scorers panel lists groups too - star one and it grades every grid cell (the aggregate in the rating pill, member verdicts as chips); starred groups are saved with Playground workbench profiles. ## Scoring live traffic Give a group an `online` profile and it scores live traffic exactly the way an online evaluator does: a group score below the alert threshold raises a Signal, events key on `scorer-group:`, and the score history feeds the group's ratings chart. The profile's `scope` picks the unit of judgment - `"trace"` (the default) scores each sampled trace as it arrives, `"session"` scores whole multi-turn conversations once they go quiet. ### Scoring each trace The default scope - every sampled ingested trace runs every member: ```python theme={null} client.monitor.scorer_groups.update(group.id, online={ "enabled": True, "sampleRate": 0.2, # 20% of traffic "alertThreshold": 6, # Signal when the GROUP score lands below 6/10 "severity": "high", # omitted keys keep their stored value - send "scope": "trace" to flip a session group back }) client.monitor.scorer_groups.ratings(group.id, window="7d") # score history for the chart ``` Live scoring has full parity with online evaluators: * **Score chip**: when a group has scored a trace, the group score **owns the Score chip** in Live Traces (the composed final opinion beats any individual verdict for the headline); individual verdicts stay in the expanded row. * **Trace Details**: when both kinds scored the trace, Trace Details shows a **Group score / Judge scores** switch - the group aggregate with its blend breakdown on one side, per-member verdicts (recorded individually at scoring time) on the other. With only one kind present, the section shows it directly, no toggle. * **Webhooks**: a below-threshold group score pages the agent's alert channels exactly like a matched pattern or a low online-evaluator verdict. * **Judge budget**: judge members draw from the same online judge-call budget online evaluators reserve from - a group with 3 judge members takes 3 slots per sampled trace, and an exhausted budget skips the group rather than scoring it on a partial panel. * **KPIs**: group scores (and member verdicts) are ratings, not runs - they never inflate the Overview's run counts. Judge members are real LLM spend at the sample rate - the sample dial is the cost dial. ### Scoring whole sessions Some failures don't live in any single reply - the agent contradicts turn 1 on turn 3, or the conversation ends without the user getting what they came for. Set `scope: "session"` and the group judges **the whole conversation as one unit**, on the same idle trigger [session evaluation](/monitor/session-evaluation) uses: once a session with 2+ turns has been quiet for `idleSeconds` (default 120), the background sweep scores it - and re-scores it automatically if the conversation later grows. Individual traces of a session-scoped group are deliberately **not** scored at ingest, so a conversation is never double-judged. ```python theme={null} client.monitor.scorer_groups.update(group.id, online={ "enabled": True, "sampleRate": 1.0, # of eligible idle sessions "scope": "session", "idleSeconds": 120, # judge once quiet for 2 minutes "alertThreshold": 6, # Signal when the conversation's GROUP score lands below 6/10 "severity": "high", }) ``` Every member reads the same assembled transcript: * **Judge members** are asked the structured whole-session question built from their criteria (consistency across turns, whether the user's need was resolved). A judge's per-trace `judgePrompt` never applies at session scope - the same rule session evaluators follow. * **Pattern and code/external members** get the conversation split by speaker, not the formatted transcript: every user turn joined together as their input, every agent turn joined together as their output - so a "response contains" condition reads everything the agent said across the conversation without ever firing on the user's own words. Both sides are read from root spans first, with fallbacks for OTel-style sessions whose roots are empty: the user side falls back to the full formatted transcript (over-matching is the honest direction for a tripwire), while the agent side falls back only to child span outputs - genuinely the agent's words - and stays empty when even those are missing. Tool calls are not passed, and a code scorer's `spans` array covers only the session's last (anchor) trace, not every turn. * Weights, renormalization, and must-pass gates work identically - an apology tripwire pattern with `weight: 0, gate: true` zeroes a conversation that apologizes anywhere. The verdict lands in the session's judge rail (Observe → Sessions, labeled "*Name* (group)"), joins the group's ratings history, and a below-threshold score raises a Signal keyed on the same `scorer-group:` - and pages the agent's webhook channels exactly like a below-threshold trace score. Judge members draw from the same online judge-call budget as trace scoring - one slot per judge call actually made. `sampleRate` is a true per-conversation fraction: the keep-or-skip decision is made once per (conversation, group) and stays stable across sweep ticks, so 0.2 really means about a fifth of conversations get judged. Reading verdicts back from the SDK: ```python theme={null} client.monitor.sessions.scores(session_id) # -> [{"kind": "scorer-group:", "rating": 4.5, "justification": "Weighted blend: ..."}, ...] ``` ## SDK surface `client.monitor.scorer_groups`: `list()`, `get(id)`, `create(name, members, description=None, online=None)`, `update(id, **fields)` (sparse; `online` is merged key-by-key with the stored profile, so omitted keys keep their values, and `online=None` detaches live scoring), `delete(id)`, and `ratings(id, window="7d")`. The key-by-key merge only works when the group already has a stored online profile. Patching `online` on a group that has none returns a 400 asking for a full online object (`enabled`, `sampleRate`, `alertThreshold`, `severity`; `scope` defaults to `"trace"`) - a partial patch against nothing would mean the engine silently inventing defaults you never chose. Limits enforced at save time: 1-20 members per group, `weight` 0-1000, `sampleRate` 0-1, `alertThreshold` 0-10 (or null), `idleSeconds` 0-86400. Members are validated at create/update time: an unknown `refId` returns a 400 instead of storing a group that silently blends nothing (refs that were already stored before a scorer was deleted are grandfathered - they degrade at scoring time as described above). A judge scorer with `requiresExpected` needs a reference answer and cannot grade live traffic, so it is rejected from any group whose live scoring is enabled - disable `requiresExpected` or keep the group's live scoring off. The runnable, assertion-style walkthroughs are `sample-scripts/eval_deep_dive/09_scorer_groups.py` (dataset runs, gates, trace-scope live scoring) and `10_session_group_scoring.py` (session scope end to end). Groups replace the older habit of chaining scorers inside one judge's [attached code checks](/evaluation/code-scorers): those still work (and can still blend via `scores`), but a group is reusable, visible in the catalog, and composes every scorer kind - not just code.