Multi-Turn Session Evaluation

Per-trace scoring answers “was this response good?”. Plenty of conversational failures are invisible at that granularity: every individual reply looks fine, but the agent contradicts what it said three turns ago, asks for the order number twice, drifts off the customer’s goal, or never actually resolves anything. Session evaluation judges the assembled multi-turn conversation as one unit, so criteria like “did the conversation resolve the customer’s problem” become scoreable. Any traces sharing a session_id form a session - pass it to the tracer, or let framework integrations set it:

with client.tracer.trace("support-agent", session_id="conv-812", input=user_msg) as span:
    ...

When does a conversation get judged?

Conversations have no end event - the engine can’t know a user won’t come back. Session evaluation answers this the way trace/group-scoped online scoring tools converged on: judge once the session has been quiet. A background sweep (every 60s) finds sessions idle longer than the evaluator’s idle_seconds (default 120) and judges the full transcript. If the conversation later resumes, the next sweep re-judges it automatically - a session whose latest score is newer than its last activity is never judged twice. Bounds that keep this safe to leave on:

Only sessions with 2+ turns qualify - single-trace sessions are already covered by per-trace scoring, and judging them again at conversation prices would double the bill.
At most 5 sessions per sweep tick are judged (a budget shared across all projects on the instance, round-robin); anything left over is picked up next tick.
The sweep looks back 24h for candidates by default. AGENTX_SESSION_SWEEP_WINDOW_HOURS widens that (values are bucketed: up to 24 means 24h, up to 168 means 7d, above that 30d) - useful after downtime, or for traffic that arrives in delayed batches.
AGENTX_SESSION_SWEEP=false disables the sweep entirely.

The built-in: Session Baseline Judge

Every project ships with one built-in session evaluator: the Session Baseline Judge. Like every scorer it is created paused - switch it on from the Scorers page. It scores whole-conversation consistency - context retention, self-contradiction, goal drift, non-repetition - and points at the first span where the conversation broke. Its rubric is an ordinary judge-scorer rubric, so Tune judge improves it from calibration evidence like any other evaluator; the row itself can be paused but not deleted. Alongside the 0-10 score, every session judgment returns structured findings: up to six per-step citations, each with a short category tag, rendered as the judge rail in the session’s detail view with the cited turns flagged inline.

Your own session criteria

Create a session-scoped evaluator exactly like a per-trace one - the only difference is scope="session":

client.monitor.judge_scorers.builder(
    "Conversation resolution",
    acceptance_criteria=(
        "The conversation ends with the customer's problem resolved or a concrete, "
        "dated next step. No information the customer already gave is asked for again."
    ),
    rejection_criteria=(
        "The agent contradicts an earlier turn, loops on clarifying questions, "
        "or the customer gives up."
    ),
    live=True,
    scope="session",
    idle_seconds=120,
    alert_threshold=5,       # a failing conversation raises a signal
    severity="high",
).publish()

A verdict below alert_threshold raises a signal in the normal triage queue, deduped like every other signal source. Verdicts also appear on the session’s detail view and in the Session score column of Observe → Sessions. Need more than one opinion per conversation? A scorer group can be session-scoped too (online: {"scope": "session", ...}): several judges, patterns, and code scorers read the same transcript and blend into one 0-10 verdict on the same idle trigger, with must-pass gates - e.g. an apology tripwire that zeroes the conversation’s score.

On demand from the SDK

The sweep is the automatic path; both checks also run on demand - the same judging, same score shape, and an explicit call ignores the evaluator’s paused state (a human asking for a score should get one):

score = client.monitor.sessions.coherence_check(session_id)   # one judge call
print(f"{score['rating']}/10 across {score['spanCount']} spans")
print(score["justification"])

if score["driftSpanId"]:
    # The drift span can be any child in the session - walk it up to its turn root.
    spans = client.monitor.sessions.spans(session_id)
    by_id = {s.get("spanId"): s for s in spans if s.get("spanId")}
    drift = next((s for s in spans if s["_id"] == score["driftSpanId"]), None)
    while drift and drift.get("parentSpanId"):
        drift = by_id.get(drift["parentSpanId"])

Two more helpers round out the surface:

# Every session-level verdict, newest first. `kind` says who scored: "online-eval:<id>"
# for a session evaluator, "scorer-group:<id>" for a scorer group, or legacy "coherence"
# rows from before the Session Baseline Judge existed.
for s in client.monitor.sessions.scores(session_id):
    print(s["kind"], s["rating"], s["justification"])

# Run the idle-session sweep once, right now, over THIS project's idle sessions (it runs
# automatically every minute) - useful in demos, tests, and backfills where waiting a tick
# is the wrong UX. Concurrent calls are serialized: a sweep already in flight returns
# {"judged": 0, "skipped": true} instead of double-judging the same sessions.
client.monitor.sessions.run_sweep()   # -> {"judged": n}

Closing the loop

A failing conversation is the best kind of regression test - multi-turn, real, and specific. Add to dataset from the session view converts the whole conversation into a dataset case (follow-up turns included, provenance kept), so the fix gets guarded by offline runs and CI gates. Simulated conversations from the Playground are recorded as real sessions too, so the same judging and the same loop apply to rehearsals before any real user is involved. The full runnable flow - weak prompt, scripted multi-turn session, per-turn and session-level judging, then prompt/tool improvement proposals from the evidence - is selfhost_demo/10_session_coherence_and_tool_improvement.py.

​When does a conversation get judged?

​The built-in: Session Baseline Judge

​Your own session criteria

​On demand from the SDK

​Closing the loop

When does a conversation get judged?

The built-in: Session Baseline Judge

Your own session criteria

On demand from the SDK

Closing the loop