Skip to main content
The Custom Agent Evaluation SDK lets you run your agent against a dataset of test questions and get back LLM-judged scores, similarity metrics, and an AI-generated analysis report. It works with any agent you own: LangChain, CrewAI, AutoGen/AG2, LlamaIndex, OpenAI, Anthropic, an HTTP endpoint, or plain Python. Unlike CI/CD evaluation, which is built for a fast pass/fail gate in a pipeline, Custom Agent Evaluations are built for depth: multi-run consistency checks, similarity scoring, and a full report with strengths, weaknesses, and recommendations.

​
How it works

  1. Build a dataset: create test cases with queries and (optionally) expected results.
  2. Run your agent: the SDK calls your function or adapter once per test case; each response is scored immediately.
  3. Finalize: closes the run. average_rating, min_rating, max_rating, and rated_count are already available at this point; no need to wait for analysis just to see how it went.
  4. Analyze (optional): AgentX generates the full qualitative report, covering strengths, weaknesses, instruction adherence, and recommendations.
  5. Review results: read the report in your terminal or open the dashboard link.

​
Installation

See Installation for framework-specific extras.

​
What’s included

​
Reusable grading configs

A dataset’s questions and its grading config (acceptance criteria, similarity metrics, thresholds) are independent, so the same config can grade any dataset. Build one once with client.evaluations.settings.builder(...).publish() and pass its id as evaluation_settings_id to client.evaluations.run(...). Omit it to use the dataset’s own bundled config, unchanged. This is purely additive. See Evaluation Settings for the full builder reference.

​
Guides

Installation

Framework-specific package extras

Quick Start

Run your first evaluation in a few lines

Build Dataset

Create test cases programmatically or from CSV

Evaluation Settings

Build a standalone, reusable grading config

EvaluationSubject Fields

Describe the agent being evaluated

AI Analysis Report

What .analyze() returns: strengths, weaknesses, recommendations

Examples

Framework-by-framework integration examples

Tracing

Attach a full Execution Timeline to each eval result, not just a score

CI/CD Evaluation

Gate releases with a pass/fail threshold instead