OpenAI Agents SDK

This is a different package from plain openai: the Agents SDK (import agents) adds its own Agent/Runner abstraction, tool calling, and handoffs on top of the Chat Completions/Responses APIs. Use this page if your code imports from agents; use the OpenAI page if you’re calling openai.chat.completions.create() directly. Install:

pip install agentx-python openai-agents

Usage

from agentx import AgentX
from agents import Agent, Runner, function_tool

client = AgentX.from_env()

@function_tool
def get_policy(topic: str) -> str:
    """Look up a company policy by topic."""
    db = {
        "cancel": "Go to Account → Subscription → Cancel.",
        "refund": "Full refund within 30 days.",
    }
    return db.get(topic.lower(), "No policy found.")

agent = Agent(
    name="support-agent",
    instructions="You are a helpful support agent. Use get_policy to look up policies.",
    tools=[get_policy],
)

def openai_agents_eval(case):
    result = Runner.run_sync(agent, case.query)
    return {"output": result.final_output}

run_context = (
    client.evaluations
    .run(dataset_id="...", subject={"kind": "custom_agent", "displayName": "Support Agent", "framework": "openai-agents"})
    .execute(openai_agents_eval)
    .finalize()
)
print("Average rating:", run_context.average_rating)

subject.framework is an open string - any label is accepted. The values listed on EvaluationSubject Fields are the conventions the dashboard groups by, and the tracing integrations stamp "openai-agents" too, so using it here keeps eval runs and traces under the same label.

With tracing

Unlike Anthropic/LangChain/CrewAI, there’s no way to get trace_id back from the Agents SDK’s own tracing integration (AgentXTracingProcessor). It’s a processor you register once at startup and it reports on its own schedule as runs complete elsewhere in the SDK, not a span your eval function controls. To link a trace to an eval result, wrap Runner.run_sync(...) in tracer.trace(..., sync=True) directly instead, the same pattern as the raw OpenAI example. Inside .execute(), traces are automatically stamped monitor=False (and source="eval-run"), so no flag is needed:

from agentx import AgentX
from agents import Agent, Runner

client = AgentX.from_env()

agent = Agent(name="support-agent", instructions="You are a helpful support agent.")

def openai_agents_eval(case):
    with client.tracer.trace(
        "openai-agents-call",
        input={"query": case.query},
        framework="openai-agents",
        sync=True,
    ) as span:
        result = Runner.run_sync(agent, case.query)
        span.output = result.final_output

    return {
        "output": result.final_output,
        "trace_id": span.trace_id,
    }

If you also want every other (non-eval) run of this agent traced automatically, not just the ones going through an evaluation, register AgentXTracingProcessor globally in addition to the pattern above. The two don’t conflict, but they are independent: the processor’s traces are separate from, and won’t be linked to, any eval result’s trace_id.

`EvaluationCase` fields

Same contract as every other framework: see Examples overview for the full EvaluationCase/return-value reference. case.query is what you pass to Runner.run_sync(agent, case.query).

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Playground

Framework Examples

OpenAI Agents SDK

Usage

With tracing

`EvaluationCase` fields

​Usage

​With tracing

​EvaluationCase fields

Usage

With tracing

`EvaluationCase` fields