Evaluation Settings

A dataset’s questions and its grading config (acceptance criteria, similarity metrics, thresholds) are independent. EvaluationSettings is the grading config half on its own: build it once, then run it against any dataset by id, instead of every dataset carrying its own copy. Since the judge-scorer unification, an evaluation-settings record IS the judge rubric + offline profile of an LLM Judge Scorer - the unified entity at client.monitor.judge_scorers, which also carries the optional online (live-traffic) profile. This surface keeps working unchanged (it now emits a default-hidden DeprecationWarning on first use); prefer judge_scorers for new code so both profiles live in one place - client.monitor.judge_scorers.builder(...) takes the same snake_case parameters as the builder below, plus tool_context, thresholds, and the live profile in the same call - and manage everything from the dashboard’s Scorers page. Passing a grader id is entirely optional (scorer_id is the current name; evaluation_settings_id is the pre-consolidation alias for the same id). Omit it and a run grades against the dataset’s own bundled config, exactly as if this feature didn’t exist. On self-host, additional_scorer_ids=[...] adds up to 4 extra judges that each score every result too - the primary keeps the rating column, extra verdicts land in judge_scorer_results per row (see the quickstart).

Programmatic builder

evaluation_settings = client.evaluations.settings.builder(
    name="Strict - production bar",
    number_of_requests=3,
    acceptance_criteria="Accurate, concise, grounded in the support policy.",
    rejection_criteria="No hallucinated policies or made-up steps.",
    vector_similarity=True,
    jaccard_similarity=True,
).publish()

print(evaluation_settings.id)

`builder()` parameters

Parameter	Type	Default	Description
`name`	`str`	required	Config display name
`description`	`str`	none	Human-readable description
`number_of_requests`	`int`	`1`	How many times each question is run per evaluation, for consistency testing
`acceptance_criteria`	`str`	none	What a good response looks like, included in the LLM scoring prompt
`rejection_criteria`	`str`	none	What a bad response looks like, included in the LLM scoring prompt
`evaluation_criteria`	`str`	none	Scoring rubric, included in the LLM scoring prompt
`vector_similarity`	`bool`	`False`	Enable cosine similarity scoring against `expected_results`
`jaccard_similarity`	`bool`	`False`	Enable Jaccard token-overlap scoring against `expected_results`
`bleu_score`	`bool`	`False`	Enable BLEU scoring against `expected_results`
`rouge_score`	`bool`	`False`	Enable ROUGE scoring against `expected_results`
`similarity_model`	`str`	provider default	Embedding model used for vector similarity, e.g. `"text-embedding-3-small"`
`sovereignty_models`	`list[str]`	none	Enables Sovereignty & Portability; models to compare when this config runs. Discover valid ids with `client.evaluations.list_models()`
`code_scorers`	`list[dict]`	none	Deterministic JS scorers run per result alongside the judge; each entry is `{"name", "code", "enabled"}`. See Code scorers
`judge_prompt`	`str`	server default	Override the LLM-as-judge’s raw prompt template. See Configuring the judge
`judge_model`	`str`	`"gpt-5.6-luna"`	Override the LLM-as-judge’s model (any OpenAI or Anthropic id from `client.evaluations.list_models()`)

publish() sends the config to AgentX and returns an EvaluationSettings object with .id, which you pass as scorer_id (or the alias evaluation_settings_id) to client.evaluations.run(). With number_of_requests above 1, runs report per-case repetition spread as caseStatistics - see Repetition statistics.

Configuring the judge

judge_prompt/judge_model apply to every scoring path, native dashboard runs and SDK/custom-agent runs alike:

evaluation_settings = client.evaluations.settings.builder(
    name="Strict - production bar",
    judge_model="claude-opus-4-8",
    judge_prompt="""You are grading a customer support response.

**User Query:** {input}

**Agent Response:**
{output}

**Expected Results:**
{expected}

Score strictly: any missing policy detail is a failing response.""",
).publish()

judge_prompt is a raw template: {input}, {output}, and {expected} are substituted in. Everything else the judge needs (chain of thought, capabilities/references, criteria, per-question judge_guideline, delegation notes) is appended automatically after it, so a custom prompt can restructure the grading philosophy without ever losing that context. Omit judge_prompt to keep the default rubric, or judge_model to keep the default (gpt-5.6-luna).

Running against a dataset

report = (
    client.evaluations
    .run(
        dataset_id="...",
        subject={"kind": "custom_agent", "displayName": "Support Bot", "framework": "langchain", "runtime": "local"},
        evaluation_settings_id=evaluation_settings.id,
    )
    .execute(my_agent)
    .finalize()
    .analyze()
)

The same evaluation_settings_id can be reused across as many dataset_ids as you want. Build it once, grade any dataset with it going forward.

Fetching an existing config

evaluation_settings = client.evaluations.settings.get("6876eee333ccc444ddd555ff")
all_configs = client.evaluations.settings.list()

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Framework Examples

Evaluation Settings

Programmatic builder

`builder()` parameters

Configuring the judge

Running against a dataset

Fetching an existing config

Next steps

Build Dataset

Quick Start

​Programmatic builder

​builder() parameters

​Configuring the judge

​Running against a dataset

​Fetching an existing config

​Next steps

Build Dataset

Quick Start

Programmatic builder

`builder()` parameters

Configuring the judge

Running against a dataset

Fetching an existing config

Next steps