> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Turn Session Evaluation

> Judge whole conversations, not just single responses - automatically, once the conversation goes quiet

Per-trace scoring answers "was this response good?". Plenty of conversational failures are
invisible at that granularity: every individual reply looks fine, but the agent contradicts
what it said three turns ago, asks for the order number twice, drifts off the customer's goal,
or never actually resolves anything. **Session evaluation** judges the assembled multi-turn
conversation as one unit, so criteria like *"did the conversation resolve the customer's
problem"* become scoreable.

Any traces sharing a `session_id` form a [session](/trace/sessions) - pass it to the tracer,
or let framework integrations set it:

```python theme={null}
with client.tracer.trace("support-agent", session_id="conv-812", input=user_msg) as span:
    ...
```

## When does a conversation get judged?

Conversations have no end event - the engine can't know a user won't come back. Session
evaluation answers this the way trace/group-scoped online scoring tools converged on: **judge
once the session has been quiet**. A background sweep (every 60s) finds sessions idle longer
than the evaluator's `idle_seconds` (default 120) and judges the full transcript. If the
conversation later resumes, the next sweep re-judges it automatically - a session whose latest
score is newer than its last activity is never judged twice.

Bounds that keep this safe to leave on:

* Only sessions with **2+ turns** qualify - single-trace sessions are already covered by
  per-trace scoring, and judging them again at conversation prices would double the bill.
* At most **5 sessions per sweep tick** are judged; anything left over is picked up next tick.
* The sweep looks back **24h** for candidates.
* `AGENTX_SESSION_SWEEP=false` disables the sweep entirely.

## The built-in: Session Baseline Judge

Every project ships with one session evaluator already on: the **Session Baseline Judge**. It
scores whole-conversation consistency - context retention, self-contradiction, goal drift,
non-repetition - and points at the first span where the conversation broke. Its rubric is an
ordinary evaluator config, so **Tune judge** improves it from
[calibration evidence](/monitor/outcomes) like any other evaluator; the row itself can be
paused but not deleted.

Alongside the 0-10 score, every session judgment returns structured **findings**: up to six
per-step citations, each with a short category tag, rendered as the judge rail in the
session's detail view with the cited turns flagged inline.

## Your own session criteria

Create a session-scoped evaluator exactly like a per-trace one - the only difference is
`scope="session"`:

```python theme={null}
settings = client.evaluations.settings.builder(
    name="Conversation resolution",
    acceptance_criteria=(
        "The conversation ends with the customer's problem resolved or a concrete, "
        "dated next step. No information the customer already gave is asked for again."
    ),
    rejection_criteria=(
        "The agent contradicts an earlier turn, loops on clarifying questions, "
        "or the customer gives up."
    ),
).publish()

client.monitor.online_evaluators.builder(
    name="Conversation resolution",
    evaluation_settings_id=settings.id,
    scope="session",
    idle_seconds=120,
    alert_threshold=5,       # a failing conversation raises a signal
    severity="high",
).publish()
```

A verdict below `alert_threshold` raises a signal in the normal
[triage queue](/monitor/patterns), deduped like every other signal source. Verdicts also
appear on the session's detail view and as the coherence column in Observe → Sessions.

## On demand from the SDK

The sweep is the automatic path; both checks also run on demand - the same judging, same
score shape, and an explicit call ignores the evaluator's paused state (a human asking for a
score should get one):

```python theme={null}
score = client.monitor.sessions.coherence_check(session_id)   # one judge call
print(f"{score['rating']}/10 across {score['spanCount']} spans")
print(score["justification"])

if score["driftSpanId"]:
    # The drift span can be any child in the session - walk it up to its turn root.
    spans = client.monitor.sessions.spans(session_id)
    by_id = {s.get("spanId"): s for s in spans if s.get("spanId")}
    drift = next((s for s in spans if s["_id"] == score["driftSpanId"]), None)
    while drift and drift.get("parentSpanId"):
        drift = by_id.get(drift["parentSpanId"])
```

## Closing the loop

A failing conversation is the best kind of regression test - multi-turn, real, and specific.
**Add to dataset** from the session view converts the whole conversation into a dataset case
(follow-up turns included, provenance kept), so the fix gets guarded by
[offline runs](/sdk/evaluations/overview) and [CI gates](/sdk/ci-cd). Simulated conversations
from the [Playground](/evaluation/simulate-conversation) are recorded as real sessions too,
so the same judging and the same loop apply to rehearsals before any real user is involved.

The full runnable flow - weak prompt, scripted multi-turn session, per-turn and session-level
judging, then prompt/tool improvement proposals from the evidence - is
[`selfhost_demo/10_session_coherence_and_tool_improvement.py`](https://github.com/AgentX-ai/AgentX-Sample-Scripts/blob/main/selfhost_demo/10_session_coherence_and_tool_improvement.py).