> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Concepts

> Core vocabulary for working with the AgentX API

## Dataset

A **dataset** (stored as `EvaluationSettings`) is a reusable set of questions with scoring configuration. It defines:

* **Questions** — the inputs to test your agent against, with optional expected answers
* **Acceptance/rejection criteria** — plain-English rules that the LLM judge applies when scoring
* **Scoring metrics** — which combination of LLM judge, vector similarity, and Jaccard similarity to use
* **CI/CD settings** — pass rate threshold, fail fast, webhook, etc.

Datasets are versioned. When you start an evaluation run, the current dataset is snapshotted so results always reference a stable state.

***

## Question

A **question** is one test case in a dataset. It has:

* `main_question.query` — the input sent to your agent
* `main_question.expectedResults` — the ideal answer (optional; required for similarity scoring)
* `follow_up_questions` — additional follow-up prompts (for multi-turn evaluations)

A dataset with `numberOfRequests: 3` runs each question **3 times**, which helps measure consistency.

***

## Evaluation Run

An **evaluation run** (stored as `Evaluate`) is one execution of a dataset against an agent. It records:

* All submitted agent outputs and their scores
* The locked dataset version used
* Agent metadata (`evaluationSubject`)
* AI analysis (if requested)
* A gate result (for CI runs)

Runs have four statuses: `in_progress` → `completed` or `failed`.

***

## Scoring

AgentX offers three scoring methods that can be combined:

### LLM Judge (default)

An LLM evaluates the agent's output against the dataset's acceptance/rejection criteria and any `expectedResults`. Returns a **rating from 1–10** and a justification.

### Vector Similarity (opt-in)

Computes the **cosine similarity** between the agent's output and `expectedResults` using a text embedding model. Returns a float 0.0–1.0. Good for measuring semantic closeness.

### Jaccard Similarity (opt-in)

Computes **token-overlap** between the agent's output and `expectedResults`. Returns a float 0.0–1.0. Fast and model-free, good for exact-match or keyword coverage checks.

Enable these per dataset:

```json theme={null}
{ "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true } }
```

***

## Gate

A **gate** is the CI/CD pass/fail decision. A run's gate is `"pass"` when both:

1. Every per-question threshold rule passes (see [Threshold](#threshold))
2. The fraction of passing questions ≥ `ci.passRateThreshold`

Otherwise it is `"fail"`.

***

## Threshold

A **threshold** is a per-metric gate rule on a dataset — e.g. "LLM rating must be ≥ 7" or "cosine similarity must be ≥ 0.8". Threshold rules are defined in the dataset settings under `thresholds.gates`.

When a question's result violates a threshold, that question is counted as *failed* for the pass rate calculation.

***

## Pass Rate

The **pass rate** is the fraction of questions that passed all threshold gates:

```
pass_rate = passed_questions / total_questions
```

Stored as a float 0.0–1.0, displayed as a percentage (e.g. `0.875` → 87.5%).

***

## Fail Fast

When **fail fast** is enabled on a dataset, a CI run is finalized immediately on the first threshold violation. The run exits as `"fail"` without waiting for the remaining questions to be submitted. This saves time in large datasets when an early failure is decisive.

***

## Reference Agent

A **reference agent** (Robot with `isReferenceAgent: true`) is auto-created when you submit a trace via the SDK. It represents your external agent on the AgentX platform — linking its traces to the Live Traces tab and the evaluation UI without requiring the agent to run inside AgentX.

***

## Evaluation Subject

The `evaluationSubject` field on a run describes the agent being evaluated:

```json theme={null}
{
  "kind": "custom_agent",
  "displayName": "My Support Bot",
  "framework": "langchain",
  "runtime": "local",
  "agentInstructions": "You are a helpful support agent..."
}
```

When `agentInstructions` is provided, the AI analysis checks whether the agent's responses adhered to its system prompt.

***

## Sovereignty & Portability Matrix

The **sovereigntyIndex** is a per-model performance breakdown computed when results include `metadata.model`. It shows how the agent performs across different LLMs, useful for model substitution and vendor independence analysis.