Skip to main content
For any captured trace, self-host can estimate how a different model would have handled the same input - cost, latency, and a quality rating, side by side with what your agent actually returned. This is an input-only replay, not a full re-run of your agent: self-host doesn’t own your agent’s code, so it has no guaranteed system prompt, tools, or conversation history to work with - only whatever the trace itself captured.

​
What gets replayed

What actually gets replayed depends on how you instrumented tracing. Checked against the real SDK tracer:
  • The raw Anthropic client patch (agentx.integrations.anthropic) captures the full messages array verbatim - multi-turn history is included. (It doesn’t currently capture Anthropic’s separate system kwarg, so the system prompt specifically can be missing even when history isn’t.)
  • The manual API (client.tracer.trace(name, input=..., metadata=...)) is completely free-form - pass input as a proper [{"role": "user", ...}, ...] message list, or metadata={"systemPrompt": "..."}, and self-host uses exactly that.
  • The higher-level framework integrations (OpenAI Agents SDK, LangChain) flatten down to plain text by default - a trace from one of these is closer to single-turn-only unless you also pass richer metadata yourself.
Self-host makes a best-effort attempt to reconstruct a real conversation from whatever shape your trace actually has (recognizable message array → use it; plain text → single user turn; a systemPrompt/system/instructions key in metadata → prepended as the system prompt) - the same defensive, try-every-known-shape approach the OTel ingestion path already uses. Tool-calling is never reproduced, even if your trace’s metadata includes tool definitions - replaying those would mean translating an arbitrary captured schema into each candidate provider’s own tool format, real separate work this doesn’t attempt. If your trace’s metadata has tool definitions, they’re shown to you in the dialog for transparency, just not sent to candidate models.

​
Running a comparison

The comparison runs from the SDK or the REST API against any trace id (the dashboard trigger was retired, but the engine surface is unchanged):
Every model, including the original captured output, is judged with the same no-ground-truth rubric (“rate this response on its own merits, 0-10”) so ratings are directly comparable. Cost is estimated from real measured token usage against pricing you control - see below. A candidate needs its provider’s key configured on the engine; a model that fails (missing key, rate limit) never blanks out the rest of the comparison - its row comes back with a null output and the reason (including the exact key to set) in its error field. Nothing about the comparison itself is persisted - it’s computed fresh each time you run it.

​
The pricing catalog

The candidate model list and $/M-token prices are dashboard-managed, not hardcoded - click Manage models (next to the judge-model picker in Settings and in the judge scorer editor) to add, edit, or remove entries. A small curated set is seeded the first time the engine starts, so there’s something to compare against out of the box; edit or delete those freely after that, or add your own - seeded entries you remove stay removed, except three catalog staples (gpt-4o-mini, gemini-2.5-pro, gemini-2.5-flash) that are re-added on boot whenever missing. Prices are approximate and point-in-time by nature (no live pricing API exists or should exist here), so keep them updated against each provider’s current pricing page if you’re relying on the cost estimate for a real decision.

​
Cache-aware cost

This same pricing catalog also drives the estimated-cost figures on traces themselves, not just the portability comparison. When the SDK’s underlying provider call reports prompt-caching token counts - Anthropic’s cache_creation_input_tokens/cache_read_input_tokens, OpenAI’s prompt_tokens_details.cached_tokens, Gemini’s cached_content_token_count - those are captured as a subset of the trace’s input tokens and priced separately from a regular input token if you’ve set optional /Mcache−readtokens∗∗/∗∗/M cache-read tokens** / **/M cache-write tokens rates on that model in Manage models. Leave either blank and it falls back to the model’s regular input rate - cache-aware pricing is opt-in per model, and a model with no cache rates configured prices exactly as it always has.