Multi-Turn Session Evaluation

Per-trace scoring answers “was this response good?”. Plenty of conversational failures are invisible at that granularity: every individual reply looks fine, but the agent contradicts what it said three turns ago, asks for the order number twice, drifts off the customer’s goal, or never actually resolves anything. Session evaluation judges the assembled multi-turn conversation as one unit, so criteria like “did the conversation resolve the customer’s problem” become scoreable. Any traces sharing a session_id form a session - pass it to the tracer, or let framework integrations set it:

with client.tracer.trace("support-agent", session_id="conv-812", input=user_msg) as span:
    ...

When does a conversation get judged?

Conversations have no end event - the engine can’t know a user won’t come back. Session evaluation answers this the way trace/group-scoped online scoring tools converged on: judge once the session has been quiet. A background sweep (every 60s) finds sessions idle longer than the evaluator’s idle_seconds (default 120) and judges the full transcript. If the conversation later resumes, the next sweep re-judges it automatically - a session whose latest score is newer than its last activity is never judged twice. Bounds that keep this safe to leave on:

Only sessions with 2+ turns qualify - single-trace sessions are already covered by per-trace scoring, and judging them again at conversation prices would double the bill.
At most 5 sessions per sweep tick are judged; anything left over is picked up next tick.
The sweep looks back 24h for candidates.
AGENTX_SESSION_SWEEP=false disables the sweep entirely.

The built-in: Session Baseline Judge

Every project ships with one session evaluator already on: the Session Baseline Judge. It scores whole-conversation consistency - context retention, self-contradiction, goal drift, non-repetition - and points at the first span where the conversation broke. Its rubric is an ordinary evaluator config, so Tune judge improves it from calibration evidence like any other evaluator; the row itself can be paused but not deleted. Alongside the 0-10 score, every session judgment returns structured findings: up to six per-step citations, each with a short category tag, rendered as the judge rail in the session’s detail view with the cited turns flagged inline.

Your own session criteria

Create a session-scoped evaluator exactly like a per-trace one - the only difference is scope="session":

settings = client.evaluations.settings.builder(
    name="Conversation resolution",
    acceptance_criteria=(
        "The conversation ends with the customer's problem resolved or a concrete, "
        "dated next step. No information the customer already gave is asked for again."
    ),
    rejection_criteria=(
        "The agent contradicts an earlier turn, loops on clarifying questions, "
        "or the customer gives up."
    ),
).publish()

client.monitor.online_evaluators.builder(
    name="Conversation resolution",
    evaluation_settings_id=settings.id,
    scope="session",
    idle_seconds=120,
    alert_threshold=5,       # a failing conversation raises a signal
    severity="high",
).publish()

A verdict below alert_threshold raises a signal in the normal triage queue, deduped like every other signal source. Verdicts also appear on the session’s detail view and as the coherence column in Observe → Sessions.

On demand from the SDK

The sweep is the automatic path; both checks also run on demand - the same judging, same score shape, and an explicit call ignores the evaluator’s paused state (a human asking for a score should get one):

score = client.monitor.sessions.coherence_check(session_id)   # one judge call
print(f"{score['rating']}/10 across {score['spanCount']} spans")
print(score["justification"])

if score["driftSpanId"]:
    # The drift span can be any child in the session - walk it up to its turn root.
    spans = client.monitor.sessions.spans(session_id)
    by_id = {s.get("spanId"): s for s in spans if s.get("spanId")}
    drift = next((s for s in spans if s["_id"] == score["driftSpanId"]), None)
    while drift and drift.get("parentSpanId"):
        drift = by_id.get(drift["parentSpanId"])

Closing the loop

A failing conversation is the best kind of regression test - multi-turn, real, and specific. Add to dataset from the session view converts the whole conversation into a dataset case (follow-up turns included, provenance kept), so the fix gets guarded by offline runs and CI gates. Simulated conversations from the Playground are recorded as real sessions too, so the same judging and the same loop apply to rehearsals before any real user is involved. The full runnable flow - weak prompt, scripted multi-turn session, per-turn and session-level judging, then prompt/tool improvement proposals from the evidence - is selfhost_demo/10_session_coherence_and_tool_improvement.py.

Live Traffic Scoring

Ground Truth & Calibration

Multi-Turn Session Evaluation

When does a conversation get judged?

The built-in: Session Baseline Judge

Your own session criteria

On demand from the SDK

Closing the loop

​When does a conversation get judged?

​The built-in: Session Baseline Judge

​Your own session criteria

​On demand from the SDK

​Closing the loop

When does a conversation get judged?

The built-in: Session Baseline Judge

Your own session criteria

On demand from the SDK

Closing the loop