> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation Settings

> Build a standalone, reusable grading config decoupled from any one dataset

A dataset's questions and its grading config (acceptance criteria, similarity metrics, thresholds) are independent. `EvaluationSettings` is the grading config half on its own: build it once, then run it against any dataset by id, instead of every dataset carrying its own copy.

Passing `evaluation_settings_id` is entirely optional. Omit it and a run grades against the dataset's own bundled config, exactly as if this feature didn't exist.

## Programmatic builder

```python theme={null}
evaluation_settings = client.evaluations.settings.builder(
    name="Strict - production bar",
    number_of_requests=3,
    acceptance_criteria="Accurate, concise, grounded in the support policy.",
    rejection_criteria="No hallucinated policies or made-up steps.",
    vector_similarity=True,
    jaccard_similarity=True,
).publish()

print(evaluation_settings.id)
```

### `builder()` parameters

| Parameter             | Type        | Default          | Description                                                                                                                            |
| --------------------- | ----------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                | `str`       | required         | Config display name                                                                                                                    |
| `description`         | `str`       | none             | Human-readable description                                                                                                             |
| `number_of_requests`  | `int`       | `1`              | How many times each question is run per evaluation, for consistency testing                                                            |
| `acceptance_criteria` | `str`       | none             | What a good response looks like, included in the LLM scoring prompt                                                                    |
| `rejection_criteria`  | `str`       | none             | What a bad response looks like, included in the LLM scoring prompt                                                                     |
| `evaluation_criteria` | `str`       | none             | Scoring rubric, included in the LLM scoring prompt                                                                                     |
| `vector_similarity`   | `bool`      | `False`          | Enable cosine similarity scoring against `expected_results`                                                                            |
| `jaccard_similarity`  | `bool`      | `False`          | Enable Jaccard token-overlap scoring against `expected_results`                                                                        |
| `bleu_score`          | `bool`      | `False`          | Enable BLEU scoring against `expected_results`                                                                                         |
| `rouge_score`         | `bool`      | `False`          | Enable ROUGE scoring against `expected_results`                                                                                        |
| `similarity_model`    | `str`       | provider default | Embedding model used for vector similarity, e.g. `"text-embedding-3-small"`                                                            |
| `sovereignty_models`  | `list[str]` | none             | Enables Sovereignty & Portability; models to compare when this config runs. Discover valid ids with `client.evaluations.list_models()` |
| `judge_prompt`        | `str`       | server default   | Override the LLM-as-judge's raw prompt template. See [Configuring the judge](#configuring-the-judge)                                   |
| `judge_model`         | `str`       | `"gpt-5.5"`      | Override the LLM-as-judge's model (any OpenAI or Anthropic id from `client.evaluations.list_models()`)                                 |

`publish()` sends the config to AgentX and returns an `EvaluationSettings` object with `.id`, which you pass as `evaluation_settings_id` to `client.evaluations.run()`.

## Configuring the judge

`judge_prompt`/`judge_model` apply to every scoring path, native dashboard runs and SDK/custom-agent runs alike:

```python theme={null}
evaluation_settings = client.evaluations.settings.builder(
    name="Strict - production bar",
    judge_model="claude-opus-4-8",
    judge_prompt="""You are grading a customer support response.

**User Query:** {input}

**Agent Response:**
{output}

**Expected Results:**
{expected}

Score strictly: any missing policy detail is a failing response.""",
).publish()
```

`judge_prompt` is a raw template: `{input}`, `{output}`, and `{expected}` are substituted in. Everything else the judge needs (chain of thought, capabilities/references, criteria, per-question `judge_guideline`, delegation notes) is appended automatically after it, so a custom prompt can restructure the grading philosophy without ever losing that context. Omit `judge_prompt` to keep the default rubric, or `judge_model` to keep the default (`gpt-5.5`).

## Running against a dataset

```python theme={null}
report = (
    client.evaluations
    .run(
        dataset_id="...",
        subject={"kind": "custom_agent", "displayName": "Support Bot", "framework": "langchain", "runtime": "local"},
        evaluation_settings_id=evaluation_settings.id,
    )
    .execute(my_agent)
    .finalize()
    .analyze()
)
```

The same `evaluation_settings_id` can be reused across as many `dataset_id`s as you want. Build it once, grade any dataset with it going forward.

## Fetching an existing config

```python theme={null}
evaluation_settings = client.evaluations.settings.get("6876eee333ccc444ddd555ff")
all_configs = client.evaluations.settings.list()
```

## Next steps

<CardGroup cols={2}>
  <Card title="Build Dataset" icon="table" href="/sdk/evaluations/build-dataset">
    Create the test cases this config will grade
  </Card>

  <Card title="Quick Start" icon="bolt" href="/sdk/evaluations/quickstart">
    Run an evaluation end to end
  </Card>
</CardGroup>