Skip to main content
For any captured trace, self-host can estimate how a different model would have handled the same input - cost, latency, and a quality rating, side by side with what your agent actually returned. This is an input-only replay, not a full re-run of your agent: self-host doesn’t own your agent’s code, so it has no guaranteed system prompt, tools, or conversation history to work with - only whatever the trace itself captured.

​
What gets replayed

What actually gets replayed depends on how you instrumented tracing. Checked against the real SDK tracer:
  • The raw Anthropic client patch (agentx.integrations.anthropic) captures the full messages array verbatim - multi-turn history is included. (It doesn’t currently capture Anthropic’s separate system kwarg, so the system prompt specifically can be missing even when history isn’t.)
  • The manual API (client.tracer.trace(name, input=..., metadata=...)) is completely free-form - pass input as a proper [{"role": "user", ...}, ...] message list, or metadata={"systemPrompt": "..."}, and self-host uses exactly that.
  • The higher-level framework integrations (OpenAI Agents SDK, LangChain) flatten down to plain text by default - a trace from one of these is closer to single-turn-only unless you also pass richer metadata yourself.
Self-host makes a best-effort attempt to reconstruct a real conversation from whatever shape your trace actually has (recognizable message array → use it; plain text → single user turn; a systemPrompt/system/instructions key in metadata → prepended as the system prompt) - the same defensive, try-every-known-shape approach the OTel ingestion path already uses. Tool-calling is never reproduced, even if your trace’s metadata includes tool definitions - replaying those would mean translating an arbitrary captured schema into each candidate provider’s own tool format, real separate work this doesn’t attempt. If your trace’s metadata has tool definitions, they’re shown to you in the dialog for transparency, just not sent to candidate models.

​
Running a comparison

From the dashboard: open any trace’s detail view (Governance → Observe → click a trace) → Check model portability. It shows exactly what will be sent (system prompt if found, each turn, a note if it fell back to plain text) before you pick which models to compare against and run anything - nothing is sent until you click Run comparison. Every model, including the original captured output, is judged with the same no-ground-truth rubric (“rate this response on its own merits, 0-10”) so ratings are directly comparable. Cost is estimated from real measured token usage against pricing you control - see below. One model failing (bad key, rate limit) never blanks out the rest of the comparison. Nothing about the comparison itself is persisted - it’s computed fresh each time you run it, same as prompt improvement suggestions. The same comparison from the SDK, against any trace id (a candidate needs its provider’s key configured on the engine, or its row degrades gracefully with the key hint as the output):

​
The pricing catalog

The candidate model list and $/M-token prices are dashboard-managed, not hardcoded - click Manage models inside the portability dialog to add, edit, or remove entries. A small curated set (GPT-4.1/4.1-mini, GPT-4o/4o-mini, Claude Opus 4.1/Sonnet 4.5/Haiku 4.5) is seeded once the first time the engine starts, so there’s something to compare against out of the box; edit or delete any of them freely after that, or add your own - nothing you remove comes back on a restart. Prices are approximate and point-in-time by nature (no live pricing API exists or should exist here), so keep them updated against each provider’s current pricing page if you’re relying on the cost estimate for a real decision.

​
Cache-aware cost

This same pricing catalog also drives the Est. cost figure shown on every individual trace’s detail view, not just the portability comparison. When the SDK’s underlying provider call reports prompt-caching token counts - Anthropic’s cache_creation_input_tokens/cache_read_input_tokens, OpenAI’s prompt_tokens_details.cached_tokens - those are captured as a subset of the trace’s input tokens (shown as their own “Cached tokens” stat: N read · M write) and priced separately from a regular input token if you’ve set optional /Mcache−readtokens∗∗/∗∗/M cache-read tokens** / **/M cache-write tokens rates on that model in Manage models. Leave either blank and it falls back to the model’s regular input rate - cache-aware pricing is opt-in per model, and a model with no cache rates configured prices exactly as it always has.