Examples

All examples follow the same pattern: wrap your framework’s output in a function that accepts an EvaluationCase and returns a str, dict, or EvaluationResult.

from agentx.evaluations.models import EvaluationCase

def my_agent(case: EvaluationCase) -> str:
    return f"Answer to: {case.query}"

run_context = (
    client.evaluations
    .run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "My Bot", "framework": "raw_python"})
    .execute(my_agent)
    .finalize()
)
print(f"Average rating: {run_context.average_rating:.2f}")  # no .analyze() needed for this

EvaluationCase gives your function:

Field	Type	Description
`case.query`	`str`	The test question’s text
`case.case_id`	`str`	Stable id for this case, e.g. `"case-0"`
`case.question_index`	`int`	0-based index into the dataset’s `questions` array
`case.run_number`	`int`	Which repetition this is (1 through the config’s `number_of_requests`)
`case.expected_results`	`str \| None`	The question’s expected-answer text, if the dataset defines one
`case.model`	`str \| None`	Set only when Sovereignty & Portability is enabled: which comparison model this run should use

What you do inside the function is entirely up to you: call an LLM directly, invoke a LangChain chain, kick off a CrewAI crew, or hit an HTTP endpoint. What you return determines what AgentX sees:

Return type	Behavior
`str`	Used directly as the response text
`dict`	`output`/`text`/`response` for the text, plus optional `metadata`, `input_tokens`, `output_tokens`, `trace_id`, `retrieval_context` (what the agent retrieved - feeds RAG judging), `error`
`EvaluationResult`	Full manual control: set every field yourself, including `observable_trace`

Returning a plain str is enough to get scored. Add metadata: {"model": ...} when you want the Sovereignty & Portability breakdown, or trace_id when you want a full Execution Timeline attached to the result. Neither is required.

Pick your framework:

OpenAI

Plain OpenAI chat completions

OpenAI Agents SDK

Agent/Runner, tool calling, handoffs

Anthropic Claude

Claude via the Anthropic SDK

LangChain

Prompt templates and chains

CrewAI

Multi-agent crews

AutoGen / AG2

Conversable agents

LlamaIndex

RAG query engines

HTTP Endpoint

Any agent exposed as a web service

Pre-Defined Results

Score outputs you already generated

A complete, runnable version of each example lives in the AgentX-Python repository.

Pre-Deployment Quality Gate

Build & Run

Scoring & Datasets

Framework Examples

OpenAI

OpenAI Agents SDK

Anthropic Claude

LangChain

CrewAI

AutoGen / AG2

LlamaIndex

HTTP Endpoint

Pre-Defined Results