> ## Documentation Index > Fetch the complete documentation index at: https://developers.agentx.so/llms.txt > Use this file to discover all available pages before exploring further. # Offline Evaluation > Score any agent from any framework or provider against a dataset - LLM-judged, on demand, before or after you ship **Offline evaluation** catches problems before your users do. It works like unit and integration tests for agent behavior: a fixed dataset of test cases with known-good answers, run on demand - before a release, after a prompt change, in CI - with reproducibility and stability as the point. Because you control the cases, you have **ground truth**: every case can carry an expected answer, the tool calls a correct run should make, even the chunks a correct retriever should fetch, and the scoring can be as strict as an exact deterministic check or as nuanced as an LLM judge. In the dashboard this is the **Evaluate** tab. Its counterpart is [Online Evaluation](/sdk/monitor) (the Monitor tab), which scores live traffic where no ground truth exists. **Doing it well:** * **Test intermediate steps, not just final answers.** A case can pin the [expected tool-call trajectory](/sdk/evaluations/build-dataset#expected-trajectories) and the [expected retrieval context](/sdk/evaluations/build-dataset#expected-retrieval-context), so a RAG pipeline's retriever and a react agent's tool choices are graded independently of the final text. * **Mix deterministic and judged scoring.** [Code scorers](/evaluation/code-scorers), similarity metrics, and trajectory/context matching are free and exactly reproducible; the LLM judge handles what rules can't express. Low-variance checks first, judgment second. * **Build the golden dataset from production.** The best test cases are real failures - [one click turns a bad trace or signal into a dataset case](/evaluation/datasets-from-production), provenance included, so every incident becomes a permanent regression test. * **Make it a gate.** [`report.gate(fail_under=..., no_regression=True)`](/sdk/ci-cd) turns any run into a CI pass/fail, so quality regressions block the merge instead of shipping. The Custom Agent Evaluation SDK lets you run your agent against a dataset of test questions and get back LLM-judged scores, similarity metrics, and an AI-generated analysis report. It works with **any agent you own**: LangChain, CrewAI, AutoGen/AG2, LlamaIndex, OpenAI, Anthropic, an HTTP endpoint, or plain Python. Unlike [CI/CD evaluation](/sdk/ci-cd), which is built for a fast pass/fail gate in a pipeline, Custom Agent Evaluations are built for depth: multi-run consistency checks, similarity scoring, and a full report with strengths, weaknesses, and recommendations. ## How it works ``` Your agent (local) │ │ agentx-python SDK ▼ AgentX API ──► scores every response, runs similarity metrics │ ▼ Report ──► terminal output + dashboard ``` 1. **Build a dataset**: create test cases with queries and (optionally) expected results. 2. **Run your agent**: the SDK calls your function or adapter once per test case; each response is scored immediately. 3. **Finalize**: closes the run. `average_rating`, `min_rating`, `max_rating`, and `rated_count` are already available at this point; no need to wait for analysis just to see how it went. 4. **Analyze (optional)**: AgentX generates the full qualitative report, covering strengths, weaknesses, instruction adherence, and recommendations. 5. **Review results**: read the report in your terminal or open the dashboard link. ## Installation ```bash theme={null} pip install agentx-python ``` See [Installation](/sdk/evaluations/installation) for framework-specific extras. ## What's included | Module | Description | | -------------------------------------------- | ------------------------------------------------------------------------------------ | | `client.evaluations` | `EvaluationsRunner`, builds and runs evaluations against a dataset | | `client.evaluations.run()` | Starts a run, returns a chainable `EvaluationRunContext` | | `client.evaluations.datasets` | Dataset builder and CSV import | | `client.evaluations.settings` | Standalone, reusable grading config builder, decoupled from any one dataset | | `agentx.evaluations.models.EvaluationCase` | The test case passed to your agent function | | `agentx.evaluations.models.EvaluationResult` | Optional return type for full control over a result | | `agentx.evaluations.adapters` | `PrecomputedAdapter`, `HttpEndpointAdapter`: evaluate without a live Python callable | ### Reusable grading configs A dataset's questions and its grading config (acceptance criteria, similarity metrics, thresholds) are independent, so the same config can grade any dataset. Build one once with `client.evaluations.settings.builder(...).publish()` and pass its id as `evaluation_settings_id` to `client.evaluations.run(...)`. Omit it to use the dataset's own bundled config, unchanged. This is purely additive. See [Evaluation Settings](/sdk/evaluations/evaluation-settings) for the full builder reference. ## Guides Framework-specific package extras Run your first evaluation in a few lines Create test cases programmatically or from CSV Build a standalone, reusable grading config Describe the agent being evaluated What .analyze() returns: strengths, weaknesses, recommendations Framework-by-framework integration examples Attach a full Execution Timeline to each eval result, not just a score Gate releases with a pass/fail threshold instead