Skip to main content
Offline evaluation catches problems before your users do. It works like unit and integration tests for agent behavior: a fixed dataset of test cases with known-good answers, run on demand - before a release, after a prompt change, in CI - with reproducibility and stability as the point. Because you control the cases, you have ground truth: every case can carry an expected answer, the tool calls a correct run should make, even the chunks a correct retriever should fetch, and the scoring can be as strict as an exact deterministic check or as nuanced as an LLM judge. In the dashboard this is the Evaluate tab. Its counterpart is Online Evaluation (the Monitor tab), which scores live traffic where no ground truth exists. Doing it well:
  • Test intermediate steps, not just final answers. A case can pin the expected tool-call trajectory and the expected retrieval context, so a RAG pipeline’s retriever and a react agent’s tool choices are graded independently of the final text.
  • Mix deterministic and judged scoring. Code scorers, similarity metrics, and trajectory/context matching are free and exactly reproducible; the LLM judge handles what rules can’t express. Low-variance checks first, judgment second.
  • Build the golden dataset from production. The best test cases are real failures - one click turns a bad trace or signal into a dataset case, provenance included, so every incident becomes a permanent regression test.
  • Make it a gate. report.gate(fail_under=..., no_regression=True) turns any run into a CI pass/fail, so quality regressions block the merge instead of shipping.
The Custom Agent Evaluation SDK lets you run your agent against a dataset of test questions and get back LLM-judged scores, similarity metrics, and an AI-generated analysis report. It works with any agent you own: LangChain, CrewAI, AutoGen/AG2, LlamaIndex, OpenAI, Anthropic, an HTTP endpoint, or plain Python. Unlike CI/CD evaluation, which is built for a fast pass/fail gate in a pipeline, Custom Agent Evaluations are built for depth: multi-run consistency checks, similarity scoring, and a full report with strengths, weaknesses, and recommendations.

​
How it works

  1. Build a dataset: create test cases with queries and (optionally) expected results.
  2. Run your agent: the SDK calls your function or adapter once per test case; each response is scored immediately.
  3. Finalize: closes the run. average_rating, min_rating, max_rating, and rated_count are already available at this point; no need to wait for analysis just to see how it went.
  4. Analyze (optional): AgentX generates the full qualitative report, covering strengths, weaknesses, instruction adherence, and recommendations.
  5. Review results: read the report in your terminal or open the dashboard link.

​
Installation

See Installation for framework-specific extras.

​
What’s included

​
Reusable grading configs

A dataset’s questions and its grading config (acceptance criteria, similarity metrics, thresholds) are independent, so the same config can grade any dataset. Build one once with client.evaluations.settings.builder(...).publish() and pass its id as evaluation_settings_id to client.evaluations.run(...). Omit it to use the dataset’s own bundled config, unchanged. This is purely additive. See Evaluation Settings for the full builder reference.

​
Guides

Installation

Framework-specific package extras

Quick Start

Run your first evaluation in a few lines

Build Dataset

Create test cases programmatically or from CSV

Evaluation Settings

Build a standalone, reusable grading config

EvaluationSubject Fields

Describe the agent being evaluated

AI Analysis Report

What .analyze() returns: strengths, weaknesses, recommendations

Examples

Framework-by-framework integration examples

Tracing

Attach a full Execution Timeline to each eval result, not just a score

CI/CD Evaluation

Gate releases with a pass/fail threshold instead