RAG Evaluation

RAG failures split into two kinds - the retriever fetched the wrong chunks, or the generator ignored the right ones - and a single end-to-end score can’t tell you which. AgentX ships the standard component-split metrics as seeded evaluator configs (Evaluate → Evaluator), usable online (live traffic) and offline (dataset runs) with the same configs.

The metric pack

Metric	Judges	A low score means
RAG: Context Relevancy	Are the chunks on-topic and sufficient for the query? (ignores the response)	Fix chunking / embeddings / top-K
RAG: Contextual Precision	Are the relevant chunks ranked above the irrelevant ones?	Fix the reranker
RAG: Contextual Recall	Does the context cover what the expected answer needs? (offline - needs a reference)	Retrieval misses source material
RAG: Faithfulness	Is every claim in the response grounded in the chunks?	The generator hallucinates past its context
RAG: Answer Relevancy	Does the response actually address the query?	Fix the prompt

All five are ordinary evaluator configs: tune their criteria, reference them from online evaluators, or pass one as evaluation_settings_id on a run. Deleted ones stay deleted; new-in-a-release metrics seed once per instance.

Where the context comes from

{context} in these judge prompts resolves through the same precedence everywhere:

Explicit - metadata.retrievalContext on a trace, or retrieval_context returned by your agent function in a run (string or list of chunk strings).
Recorded retrievals - the trace’s own retrieval spans: tracer.trace_retrieval(...), LangChain/LlamaIndex retriever callbacks, OTel retrieval spans. No caller changes needed: if your integration already records retrievals, the judges see the chunks. Custom span names are fine (trace_retrieval("kb_search", ...)) - retrieval spans carry an explicit kind marker, so they’re recognized regardless of what you name them.
Pinned (offline only) - the dataset case’s static retrievalContext.

Online: score live RAG traffic

Create an LLM judge scorer referencing a RAG config (Scorers → New scorer → LLM judge scorer, or the SDK) - every sampled trace is judged with whatever it actually retrieved:

client.monitor.online_evaluators.builder(
    name="Faithfulness (live)",
    evaluation_settings_id=faithfulness_config_id,   # from Evaluate -> Evaluator
    sample_rate=0.2,
    alert_threshold=5,        # low faithfulness raises a signal
).publish()

A response claiming a 90-day refund while the retrieved policy chunk says 30 days scores 0, with the contradiction named in the justification - and raises a signal you can send straight to a regression dataset.

Offline: dataset runs with dynamic context

A real RAG agent retrieves per query, so return what it retrieved alongside the answer - the faithfulness judge grades against that run’s chunks, not a stale pin:

def rag_agent(case):
    # Tracing the case links each result to its trace (the View Trace button in the run
    # detail table, plus trajectory-aware judging); monitor=False keeps eval traffic out
    # of online checks so nothing gets double-judged.
    with client.tracer.trace("rag-agent", input={"query": case.query}, sync=True, monitor=False) as span:
        with client.tracer.trace_retrieval("kb_search", query=case.query) as r:
            chunks = retriever.search(case.query)
            r.output = [c.text for c in chunks]
        answer = generate(case.query, chunks)
        span.output = answer
    return {
        "output": answer,
        "retrieval_context": [c.text for c in chunks],
        "trace_id": span.trace_id,
    }

client.evaluations.run(
    dataset_id=dataset.id,
    evaluation_settings_id=faithfulness_config_id,
).execute(rag_agent).finalize()

The explicit retrieval_context return is optional here - with the retrieval recorded on the linked trace, the engine pulls the chunks from its retrieval spans automatically; returning it just takes precedence. Contextual Recall additionally needs expected_results on the case (it attributes the reference answer’s claims to the context).

Deterministic: expected context match (no judge)

The five metrics above are LLM judges - each score is a model call. For retriever regression checks, a case can instead pin the chunks a correct retriever should fetch, and the engine compares them against what was actually retrieved with token-level Jaccard similarity - deterministic, free, and safe to run on every retriever, chunking, or embedding change:

dataset = (
    client.evaluations.datasets.builder(name="Retriever regression")
    .add_case(
        query="What is the refund window?",
        expected_retrieval_context=["Refunds are available within 30 days of delivery..."],
    )
    .publish()
)

The same field is editable in the dashboard: open a dataset case and fill Expected retrieval context (blank line between chunks), next to Expected tool calls. Full parameter reference: Build a dataset. Each result gets a Context match (jaccard) scorer row (0-1) next to any judge scores, computed from the same actual context the judges see (the result’s retrieval_context, else the linked trace’s retrieval spans). Jaccard measures token overlap, not meaning - use it to catch “the retriever stopped returning the right chunk”, and the judge metrics for whether the chunks are semantically right.

The loop

Capture retrievals → RAG pack scores live traffic → low scores raise signals → failing traces become dataset cases → offline runs guard the fix → CI gates keep it from regressing.

Runnable examples

Four end-to-end scripts in sample-scripts/sdk_rag_samples: online Faithfulness with automatic retrieval-span context, online Context Relevancy catching a broken retriever (and raising a high-severity signal), offline Faithfulness with dynamic per-case context, and the deterministic Jaccard retriever regression check.

Pre-Deployment Quality Gate

Build & Run

Scoring & Datasets

Framework Examples

RAG Evaluation

The metric pack

Where the context comes from

Online: score live RAG traffic

Offline: dataset runs with dynamic context

Deterministic: expected context match (no judge)

The loop

Runnable examples

​The metric pack

​Where the context comes from

​Online: score live RAG traffic

​Offline: dataset runs with dynamic context

​Deterministic: expected context match (no judge)

​The loop

​Runnable examples

The metric pack

Where the context comes from

Online: score live RAG traffic

Offline: dataset runs with dynamic context

Deterministic: expected context match (no judge)

The loop

Runnable examples