Quick Start

Prerequisites

Install the SDK.
Build a dataset, or use an existing dataset ID from the AgentX dashboard.
Export AGENTX_API_KEY in your environment.

Run an evaluation

from agentx import AgentX

client = AgentX.from_env()

def my_agent(case):
    return f"Answer to: {case.query}"

run_context = (
    client.evaluations
    .run(
        dataset_id="existing-dataset-id",
        subject={"kind": "custom_agent", "displayName": "My Agent", "framework": "raw_python"},
    )
    .execute(my_agent)
    .finalize()
)

# average_rating/min_rating/max_rating/rated_count are already available here, no need to
# call .analyze() just to see how the run went.
print(f"Average rating: {run_context.average_rating:.2f}")

report = run_context.analyze()
print(f"Dashboard: {report.dashboard_url}")

client.evaluations.run() creates the run, .execute() calls my_agent once per test case, scores each response immediately, and submits the results. .finalize() closes the run: run_context.average_rating (and .min_rating / .max_rating / .rated_count) read straight from those scores, no LLM analysis pass required. .analyze() is a separate, optional step that adds the qualitative report (strengths, weaknesses, recommendations) and returns it as report.

Link each result to its trace

Trace the agent call and return the span’s id alongside the output - three things light up at once: a View trace button on every result row, trajectory-aware judging (the judge sees the tools the agent actually called, not just the answer), and expected-trajectory matching for cases that declare expected_tools:

def my_agent(case):
    # sync=True populates span.trace_id before the block exits; monitor=False keeps the run's
    # own judge as the only scorer (no double-judging at ingest).
    with client.tracer.trace(
        "my-agent", input={"query": case.query}, sync=True, monitor=False
    ) as span:
        answer = run_my_agent(case.query)
        span.output = answer
    return {"output": answer, "trace_id": span.trace_id}

The dashboard’s Create evaluation button (Evaluate → Runs) generates this exact scaffold with your real dataset and evaluator ids inlined, in Simple, OpenAI, LangChain, or Google ADK flavors.

Concurrency, output reuse, and resume

.execute() takes two optional knobs:

run_context = (
    client.evaluations
    .run(dataset_id="existing-dataset-id", subject=subject)
    .execute(my_agent, concurrency=4)               # cases run in a thread pool
    .finalize()
)

# Iterating on grading config? Replay the previous run's recorded outputs instead of
# calling the agent again - unchanged cases are re-scored with THIS run's grading config.
run_context = (
    client.evaluations
    .run(dataset_id="existing-dataset-id", subject=subject, scorer_id=stricter_scorer.id)
    .execute(my_agent, reuse_outputs_from="previous-run-id")
    .finalize()
)

concurrency (default 1) bounds parallel agent calls; results are still submitted in case order. reuse_outputs_from replays the recorded outputs of cases unchanged since that run and only invokes your agent for new or edited ones - scorer iteration costs judge calls, not agent calls. Replayed results carry reusedFromRun in their metadata. Interrupted runs resume for free: the engine tracks which case results a run already received, and re-running .execute() against the same run skips them (older engines without the route just re-run everything). A batch that fails to submit is retried once and then raises EvaluationSubmissionError instead of letting the run finish silently empty - the run is left unfinalized, and re-running execute() picks up past the already-submitted cases.

Configuration

AgentX reads environment variables via .from_env():

Variable	Description
`AGENTX_API_KEY`	Workspace API key (required)
`AGENTX_API_BASE_URL`	Optional override for the API base URL
`AGENTX_WORKSPACE_ID`	Explicit workspace ID. Set this if your API key’s user belongs to more than one workspace, otherwise datasets, configs, and runs are created in whichever workspace the key defaults to, which may not be the one you intend.

Or pass configuration directly:

client = AgentX(
    api_key="your-key",
    base_url="https://your-agentx-instance.com/api/v1",  # include the /api/v1 suffix
    workspace_id="your-workspace-id",
)

Next steps

Build Dataset

Create test cases instead of using an existing dataset

EvaluationSubject Fields

Describe your agent so analysis can check instruction adherence

Examples

Framework-specific integration patterns

Submit Pre-Defined Results

Score outputs you already generated, without re-running the agent

Evaluation Settings

Build one grading config once, run it against any dataset

Link a Trace to a Result

Get a full Execution Timeline per result, not just a score

Multiple judge scorers per run

One agent execution, N verdicts (self-host): pass additional_scorer_ids and every result is also scored by each extra judge, with its own rubric and model. The primary scorer keeps the rating column (gates and averages unchanged); extra verdicts land in each result row’s judgeScorerResults and roll up into the run’s scorerBreakdown.

run = client.evaluations.run(
    dataset_id,
    subject,
    scorer_id=quality.id,
    additional_scorer_ids=[safety.id, tone.id],
).execute(my_agent).finalize()

for row in run.results():
    extras = {v["name"]: v["rating"] for v in (row.judge_scorer_results or [])}
    print(row.rating, extras)  # e.g. 9.0 {'Safety': 3.0, 'Tone': 6.0}

The CI gate can target a named additional scorer - “fail if Safety is low even when the average looks fine”:

gate = run.gate(fail_under=8, scorer="Safety")  # gates Safety's own average
print(gate.passed, gate.gated_scorer)           # e.g. False {'id': '...', 'name': 'Safety'}

(Equivalent REST: GET /custom-agent-evaluations/runs/{id}/gate?failUnder=8&scorer=Safety; an unknown scorer id or name is a hard 400, never a silently-passing gate.)

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Framework Examples

Prerequisites

Run an evaluation

Link each result to its trace

Concurrency, output reuse, and resume

Configuration

Next steps

Build Dataset

EvaluationSubject Fields

Examples

Submit Pre-Defined Results

Evaluation Settings

Link a Trace to a Result

Multiple judge scorers per run

​Prerequisites

​Run an evaluation

​Link each result to its trace

​Concurrency, output reuse, and resume

​Configuration

​Next steps

Build Dataset

EvaluationSubject Fields

Examples

Submit Pre-Defined Results

Evaluation Settings

Link a Trace to a Result

​Multiple judge scorers per run

Prerequisites

Run an evaluation

Link each result to its trace

Concurrency, output reuse, and resume

Configuration

Next steps

Multiple judge scorers per run