EvaluationSubject Fields

subject (passed to client.evaluations.run()) tells AgentX what it’s evaluating. It’s used by AI analysis to check instruction adherence and to power the Sovereignty & Portability matrix; it never changes how scoring itself runs.

report = (
    client.evaluations
    .run(
        dataset_id="...",
        subject={
            "kind": "custom_agent",
            "displayName": "Support Bot",
            "framework": "langchain",
            "runtime": "local",
            "agentInstructions": "You are a helpful support agent...",
        },
    )
    .execute(my_agent)
    .finalize()
    .analyze()
)

Fields

Field	Values	Description
`kind`	`"custom_agent"`	Always `"custom_agent"` for externally-owned agents evaluated via this SDK
`displayName`	`str`	Human-readable identifier shown in the dashboard
`framework`	`"raw_python"` \| `"openai"` \| `"anthropic"` \| `"google"` \| `"langchain"` \| `"llamaindex"` \| `"crewai"` \| `"autogen"` \| `"n8n"` \| `"flowise"` \| `"other"`	Underlying technology stack
`runtime`	`"local"` \| `"ci"` \| `"customer_hosted"` \| `"low_code"`	Where the agent is running. Defaults to `"local"`
`frameworkVersion`	`str`	Optional framework version identifier, e.g. `"0.3.14"`
`agentInstructions`	`str`	Optional system prompt / instructions, checked against actual behavior during analysis
`metadata`	`dict[str, str \| int \| bool]`	Optional free-form tags recorded on the run

Every field is optional: kind defaults to "custom_agent" and runtime to "local". Set at least displayName and framework so the dashboard can label the run; use "other" when your stack isn’t in the framework list (any value outside it is rejected - see the note on the OpenAI Agents SDK example). Snake_case keys (display_name, agent_instructions, …) are accepted too.

Return values from your agent function

Whatever callable you pass to .execute() can return:

A string: used directly as the output text.
A dict: "output" (or "text"/"response") is the output text. Recognized optional keys: "trace_id" links the result to a recorded trace (enabling View trace, trajectory-aware judging, and expected-trajectory matching); "retrieval_context" (string or chunk list) feeds RAG judging; "metadata" (dict) is stored on the result; "input_tokens"/"output_tokens" fill in the timing detail; "trace" attaches a lightweight {"events": [...]} observable trace; "error" marks the case as failed.
An EvaluationResult: full manual control, including the observable trace and timings. Case identity fields are required by the model (the SDK re-stamps them from the case anyway):

from agentx.evaluations.models import EvaluationResult, ResultTimings

def my_agent(case):
    return EvaluationResult(
        case_id=case.case_id,
        question_index=case.question_index,
        run_number=case.run_number,
        input={"query": case.query},
        output={"text": "Click Forgot Password on the login screen."},
        metadata={"model": "gpt-4o"},
        timings=ResultTimings(latency_ms=1240, input_tokens=150, output_tokens=45),
    )

For most agents the dict form is all you need; reach for EvaluationResult only when you want to hand-build the observable trace or error object.

The SDK uploads only what your callable returns, plus the case’s query as the result’s input. Nothing else from your process (environment, prompts you didn’t return, other variables) is sent.

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Framework Examples

EvaluationSubject Fields

Fields

Return values from your agent function

​Fields

​Return values from your agent function

Fields

Return values from your agent function