Skip to main content

​
Dataset

A dataset (stored as EvaluationSettings) is a reusable set of questions with scoring configuration. It defines:
  • Questions — the inputs to test your agent against, with optional expected answers
  • Acceptance/rejection criteria — plain-English rules that the LLM judge applies when scoring
  • Scoring metrics — which combination of LLM judge, vector similarity, and Jaccard similarity to use
  • CI/CD settings — pass rate threshold, fail fast, webhook, etc.
Datasets are versioned. When you start an evaluation run, the current dataset is snapshotted so results always reference a stable state.

​
Question

A question is one test case in a dataset. It has:
  • main_question.query — the input sent to your agent
  • main_question.expectedResults — the ideal answer (optional; required for similarity scoring)
  • follow_up_questions — additional follow-up prompts (for multi-turn evaluations)
A dataset with numberOfRequests: 3 runs each question 3 times, which helps measure consistency.

​
Evaluation Run

An evaluation run (stored as Evaluate) is one execution of a dataset against an agent. It records:
  • All submitted agent outputs and their scores
  • The locked dataset version used
  • Agent metadata (evaluationSubject)
  • AI analysis (if requested)
  • A gate result (for CI runs)
Runs have four statuses: in_progress → completed or failed.

​
Scoring

AgentX offers three scoring methods that can be combined:

​
LLM Judge (default)

An LLM evaluates the agent’s output against the dataset’s acceptance/rejection criteria and any expectedResults. Returns a rating from 1–10 and a justification.

​
Vector Similarity (opt-in)

Computes the cosine similarity between the agent’s output and expectedResults using a text embedding model. Returns a float 0.0–1.0. Good for measuring semantic closeness.

​
Jaccard Similarity (opt-in)

Computes token-overlap between the agent’s output and expectedResults. Returns a float 0.0–1.0. Fast and model-free, good for exact-match or keyword coverage checks. Enable these per dataset: 
{ "vectorSimilarity": { "enabled": true }, "jaccardSimilarity": { "enabled": true } }

​
Gate

A gate is the CI/CD pass/fail decision. A run’s gate is "pass" when both:
  1. Every per-question threshold rule passes (see Threshold)
  2. The fraction of passing questions ≥ ci.passRateThreshold
Otherwise it is "fail".

​
Threshold

A threshold is a per-metric gate rule on a dataset — e.g. “LLM rating must be ≥ 7” or “cosine similarity must be ≥ 0.8”. Threshold rules are defined in the dataset settings under thresholds.gates. When a question’s result violates a threshold, that question is counted as failed for the pass rate calculation.

​
Pass Rate

The pass rate is the fraction of questions that passed all threshold gates: 
pass_rate = passed_questions / total_questions
Stored as a float 0.0–1.0, displayed as a percentage (e.g. 0.875 → 87.5%).

​
Fail Fast

When fail fast is enabled on a dataset, a CI run is finalized immediately on the first threshold violation. The run exits as "fail" without waiting for the remaining questions to be submitted. This saves time in large datasets when an early failure is decisive.

​
Reference Agent

A reference agent (Robot with isReferenceAgent: true) is auto-created when you submit a trace via the SDK. It represents your external agent on the AgentX platform — linking its traces to the Live Traces tab and the evaluation UI without requiring the agent to run inside AgentX.

​
Evaluation Subject

The evaluationSubject field on a run describes the agent being evaluated: 
{
  "kind": "custom_agent",
  "displayName": "My Support Bot",
  "framework": "langchain",
  "runtime": "local",
  "agentInstructions": "You are a helpful support agent..."
}
When agentInstructions is provided, the AI analysis checks whether the agent’s responses adhered to its system prompt.

​
Sovereignty & Portability Matrix

The sovereigntyIndex is a per-model performance breakdown computed when results include metadata.model. It shows how the agent performs across different LLMs, useful for model substitution and vendor independence analysis.