> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG Evaluation

> The five RAG metrics, online and offline, with context captured automatically from your traces

RAG failures split into two kinds - the retriever fetched the wrong chunks, or the generator
ignored the right ones - and a single end-to-end score can't tell you which. AgentX ships the
standard component-split metrics as seeded evaluator configs (Evaluate → Evaluator), usable
**online** (live traffic) and **offline** (dataset runs) with the same configs.

## The metric pack

| Metric                        | Judges                                                                                 | A low score means                           |
| ----------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------- |
| **RAG: Context Relevancy**    | Are the chunks on-topic and sufficient for the query? (ignores the response)           | Fix chunking / embeddings / top-K           |
| **RAG: Contextual Precision** | Are the relevant chunks ranked above the irrelevant ones?                              | Fix the reranker                            |
| **RAG: Contextual Recall**    | Does the context cover what the *expected answer* needs? (offline - needs a reference) | Retrieval misses source material            |
| **RAG: Faithfulness**         | Is every claim in the response grounded in the chunks?                                 | The generator hallucinates past its context |
| **RAG: Answer Relevancy**     | Does the response actually address the query?                                          | Fix the prompt                              |

All five are ordinary evaluator configs: tune their criteria, reference them from online
evaluators, or pass one as `evaluation_settings_id` on a run. Deleted ones stay deleted;
new-in-a-release metrics seed once per instance.

## Where the context comes from

`{context}` in these judge prompts resolves through the same precedence everywhere:

1. **Explicit** - `metadata.retrievalContext` on a trace, or `retrieval_context` returned by
   your agent function in a run (string or list of chunk strings).
2. **Recorded retrievals** - the trace's own retrieval spans: `tracer.trace_retrieval(...)`,
   LangChain/LlamaIndex retriever callbacks, OTel retrieval spans. **No caller changes needed**:
   if your integration already records retrievals, the judges see the chunks. Custom span names
   are fine (`trace_retrieval("kb_search", ...)`) - retrieval spans carry an explicit kind
   marker, so they're recognized regardless of what you name them.
3. **Pinned** (offline only) - the dataset case's static `retrievalContext`.

## Online: score live RAG traffic

Create an LLM judge scorer referencing a RAG config (Scorers → New scorer → LLM judge scorer, or the SDK) -
every sampled trace is judged with whatever it actually retrieved:

```python theme={null}
client.monitor.online_evaluators.builder(
    name="Faithfulness (live)",
    evaluation_settings_id=faithfulness_config_id,   # from Evaluate -> Evaluator
    sample_rate=0.2,
    alert_threshold=5,        # low faithfulness raises a signal
).publish()
```

A response claiming a 90-day refund while the retrieved policy chunk says 30 days scores 0,
with the contradiction named in the justification - and raises a signal you can send straight
to a [regression dataset](/evaluation/datasets-from-production).

## Offline: dataset runs with dynamic context

A real RAG agent retrieves per query, so return what it retrieved alongside the answer - the
faithfulness judge grades against *that run's* chunks, not a stale pin:

```python theme={null}
def rag_agent(case):
    # Tracing the case links each result to its trace (the View Trace button in the run
    # detail table, plus trajectory-aware judging); monitor=False keeps eval traffic out
    # of online checks so nothing gets double-judged.
    with client.tracer.trace("rag-agent", input={"query": case.query}, sync=True, monitor=False) as span:
        with client.tracer.trace_retrieval("kb_search", query=case.query) as r:
            chunks = retriever.search(case.query)
            r.output = [c.text for c in chunks]
        answer = generate(case.query, chunks)
        span.output = answer
    return {
        "output": answer,
        "retrieval_context": [c.text for c in chunks],
        "trace_id": span.trace_id,
    }

client.evaluations.run(
    dataset_id=dataset.id,
    evaluation_settings_id=faithfulness_config_id,
).execute(rag_agent).finalize()
```

The explicit `retrieval_context` return is optional here - with the retrieval recorded on the
linked trace, the engine pulls the chunks from its retrieval spans automatically; returning it
just takes precedence. **Contextual Recall** additionally needs `expected_results` on the case
(it attributes the reference answer's claims to the context).

## Deterministic: expected context match (no judge)

The five metrics above are LLM judges - each score is a model call. For retriever *regression*
checks, a case can instead pin the chunks a correct retriever should fetch, and the engine
compares them against what was actually retrieved with **token-level Jaccard similarity** -
deterministic, free, and safe to run on every retriever, chunking, or embedding change:

```python theme={null}
dataset = (
    client.evaluations.datasets.builder(name="Retriever regression")
    .add_case(
        query="What is the refund window?",
        expected_retrieval_context=["Refunds are available within 30 days of delivery..."],
    )
    .publish()
)
```

The same field is editable in the dashboard: open a dataset case and fill **Expected
retrieval context** (blank line between chunks), next to Expected tool calls. Full parameter
reference: [Build a dataset](/sdk/evaluations/build-dataset#expected-retrieval-context).

Each result gets a **Context match (jaccard)** scorer row (0-1) next to any judge scores,
computed from the same actual context the judges see (the result's `retrieval_context`, else
the linked trace's retrieval spans). Jaccard measures token overlap, not meaning - use it to
catch "the retriever stopped returning the right chunk", and the judge metrics for whether
the chunks are *semantically* right.

## The loop

Capture retrievals → RAG pack scores live traffic → low scores raise signals → failing traces
become dataset cases → offline runs guard the fix → [CI gates](/sdk/ci-cd) keep it from
regressing.

## Runnable examples

Four end-to-end scripts in
[`sample-scripts/sdk_rag_samples`](https://github.com/AgentX-ai/AgentX-Sample-Scripts/tree/main/sdk_rag_samples):
online Faithfulness with automatic retrieval-span context, online Context Relevancy catching a
broken retriever (and raising a high-severity signal), offline Faithfulness with dynamic
per-case context, and the deterministic Jaccard retriever regression check.