Multi-Turn Session Evaluation

Per-trace scoring answers “was this response good?”. Plenty of conversational failures are invisible at that granularity: every individual reply looks fine, but the agent contradicts what it said three turns ago, asks for the order number twice, drifts off the customer’s goal, or never actually resolves anything. Session evaluation judges the assembled multi-turn conversation as one unit, so criteria like “did the conversation resolve the customer’s problem” become scoreable. Any traces sharing a session_id form a session - pass it to the tracer, or let framework integrations set it:

with client.tracer.trace("support-agent", session_id="conv-812", input=user_msg) as span:
    ...

When does a conversation get judged?

Conversations have no end event - the engine can’t know a user won’t come back. Session evaluation answers this the way trace/group-scoped online scoring tools converged on: judge once the session has been quiet. A background sweep (every 60s) finds sessions idle longer than the evaluator’s idle_seconds (default 120) and judges the full transcript. If the conversation later resumes, the next sweep re-judges it automatically - a session whose latest score is newer than its last activity is never judged twice. Bounds that keep this safe to leave on:

Only sessions with 2+ turns qualify - single-trace sessions are already covered by per-trace scoring, and judging them again at conversation prices would double the bill.
Each sweep tick spends at most a budget of 5 judge calls (shared across all projects on the instance, round-robin) - the budget bounds calls, not sessions, so a session-scoped scorer group with N judge members consumes N units for one session. Anything left over is picked up next tick.
The sweep looks back 24h for candidates by default. AGENTX_SESSION_SWEEP_WINDOW_HOURS widens that (values are bucketed: up to 24 means 24h, up to 168 means 7d, above that 30d) - useful after downtime, or for traffic that arrives in delayed batches.
sample_rate (builder default 0.1) is a true per-conversation fraction: the keep-or-skip decision is hashed once per (conversation, scorer) and stays stable across sweep ticks, so a sampled-out conversation stays out permanently instead of being re-rolled every minute until it eventually gets judged.
AGENTX_SESSION_SWEEP=false disables the sweep entirely.

When a session does not get scored

A scorer that fails on a session (judge model breaking the schema, every group member deleted) backs off for about 10 minutes before that (session, scorer) pair is retried, so a few stuck sessions can’t starve the healthy ones out of the tick budget.
A judge that returns no usable verdict (empty or unparseable, even after the automatic retry) stores no score row - the session stays retryable on a later tick instead of being marked fresh - and the failure is recorded as an online_eval_judge_failure event, excluded from ratings and calibration.
With multiple engine replicas sharing one database, one replica wins the sweep lease per tick and does the judging; the others skip. A crashed holder’s lease times out on its own.

The built-in: Session Baseline Judge

Every project ships with one built-in session evaluator: the Session Baseline Judge. Like every scorer it is created paused - switch it on from the Scorers page. It scores whole-conversation consistency - context retention, self-contradiction, goal drift, non-repetition - and points at the first span where the conversation broke. Its rubric is an ordinary judge-scorer rubric, so Tune judge improves it from calibration evidence like any other evaluator; the row itself can be paused but not deleted. Alongside the 0-10 score, every session judgment returns structured findings: up to six per-step citations, each with a short category tag, rendered as the judge rail in the session’s detail view with the cited turns flagged inline.

Your own session criteria

Create a session-scoped evaluator exactly like a per-trace one - the only difference is scope="session":

client.monitor.judge_scorers.builder(
    "Conversation resolution",
    acceptance_criteria=(
        "The conversation ends with the customer's problem resolved or a concrete, "
        "dated next step. No information the customer already gave is asked for again."
    ),
    rejection_criteria=(
        "The agent contradicts an earlier turn, loops on clarifying questions, "
        "or the customer gives up."
    ),
    live=True,
    scope="session",
    idle_seconds=120,
    sample_rate=1.0,         # judge every conversation (default samples 10%)
    alert_threshold=5,       # a failing conversation raises a signal
    severity="high",
).publish()

A verdict below alert_threshold raises a signal in the normal triage queue, deduped like every other signal source, and pages the agent’s webhook alert channels (the channels list on its monitor profile) - the same fan-out a below-threshold trace verdict gets. Verdicts also appear on the session’s detail view and in the Session score column of Observe → Sessions. Need more than one opinion per conversation? A scorer group can be session-scoped too (online: {"scope": "session", ...}): several judges, patterns, and code scorers read the same transcript and blend into one 0-10 verdict on the same idle trigger, with must-pass gates - e.g. an apology tripwire that zeroes the conversation’s score.

On demand from the SDK

The sweep is the automatic path; both checks also run on demand - the same judging, same score shape, and an explicit call ignores the evaluator’s paused state (a human asking for a score should get one):

score = client.monitor.sessions.coherence_check(session_id)   # one judge call
print(f"{score['rating']}/10 across {score['spanCount']} spans")
print(score["justification"])

if score["driftSpanId"]:
    # The drift span can be any child in the session - walk it up to its turn root.
    spans = client.monitor.sessions.spans(session_id)
    by_id = {s.get("spanId"): s for s in spans if s.get("spanId")}
    drift = next((s for s in spans if s["_id"] == score["driftSpanId"]), None)
    while drift and drift.get("parentSpanId"):
        drift = by_id.get(drift["parentSpanId"])

Three more helpers round out the surface:

# Every session-level verdict, newest first. `kind` says who scored: "online-eval:<id>"
# for a session evaluator, "scorer-group:<id>" for a scorer group, or legacy "coherence"
# rows from before the Session Baseline Judge existed.
for s in client.monitor.sessions.scores(session_id):
    print(s["kind"], s["rating"], s["justification"])

# Run the idle-session sweep once, right now, over THIS project's idle sessions (it runs
# automatically every minute) - useful in demos, tests, and backfills where waiting a tick
# is the wrong UX. Concurrent calls are serialized: a sweep already in flight returns
# {"judged": 0, "skipped": true} instead of double-judging the same sessions.
client.monitor.sessions.run_sweep()   # -> {"judged": n}

# On-demand verdict from any session evaluator, right now. Pass if_stale=True to skip
# the judge call when a fresh verdict already exists - it returns {"skipped": true}.
client.monitor.sessions.judge(session_id, evaluator_id, if_stale=False)

Closing the loop

A failing conversation is the best kind of regression test - multi-turn, real, and specific. Add to dataset from the session view converts the whole conversation into a dataset case (follow-up turns included, provenance kept), so the fix gets guarded by offline runs and CI gates. Simulated conversations from the Playground are recorded as real sessions too, so the same judging and the same loop apply to rehearsals before any real user is involved. The full runnable flow - weak prompt, scripted multi-turn session, per-turn and session-level judging, then prompt/tool improvement proposals from the evidence - is selfhost_demo/10_session_coherence_and_tool_improvement.py.

The Scorer Catalog

Judge Calibration & Tuning

Live Traffic Scoring

Multi-Turn Session Evaluation

When does a conversation get judged?

When a session does not get scored

The built-in: Session Baseline Judge

Your own session criteria

On demand from the SDK

Closing the loop

​When does a conversation get judged?

​When a session does not get scored

​The built-in: Session Baseline Judge

​Your own session criteria

​On demand from the SDK

​Closing the loop

When does a conversation get judged?

When a session does not get scored

The built-in: Session Baseline Judge

Your own session criteria

On demand from the SDK

Closing the loop