Skip to main content
The prompt isn’t the only text you feed an LLM that quietly decides your agent’s quality - tool definitions (the name, description, and JSON parameter schema you pass as tools=[...]) are the other half, and a vague parameter description causes malformed tool calls the same way a vague prompt causes bad answers. Tools & MCPs in the sidebar’s Manage section gives them the same registry treatment as prompts: version-scoped, edited in a schema-aware JSON editor, with full version history. Suggest improvement works the same way as a prompt’s, fed by tool-specific evidence: recorded tool calls that failed (success: false, the same field tracer.trace_tool_call captures automatically when your tool raises) and low-rated online-evaluator traces that used the tool. The judge proposes a rewritten definition - typically tightening parameter descriptions and enums against the failures it saw - shown as an editable diff, and nothing publishes until you approve it. Your agent’s code then pulls the published definition instead of hardcoding it, the same source-of-truth pattern as the prompt registry. The Playground can also add a tool straight from this registry, so you can test a proposed definition against real models before publishing it. The whole loop is scriptable via client.evaluations.tool_schemas - the tool-definition analog of the prompt registry’s SDK path:
Before publishing a proposed definition, validate it: both versions run against the tool’s own production failures (or a dataset), measured on tool choice and argument validity.

Tools you haven’t registered yet

Failure classification was never registry-gated: agent-tool-failure:<name> events tally for any traced tool (an operational outcome, not a scorer - it feeds the KPI metrics and this evidence loop directly, without raising triage signals). What registration unlocks is the improvement loop - so the Tool Schemas view has a Registered / Unregistered toggle, with live counts on each side. Unregistered lists tool names observed in recent traces that aren’t in the registry yet, with call/failure counts and a drafted definition ready to review - verbatim from the trace’s metadata.tools when available, otherwise inferred from the arguments the model actually formed (marked as inferred). The SDK’s raw-client integrations (patch_openai_client, patch_anthropic_client, the LiteLLM logger, Google GenAI) and the LangChain/LangGraph handler capture the request’s tools=[...] definitions automatically and attach them as metadata.tools - so any tool your agent actually passed to the model surfaces here with its real name, description, and parameter schema, ready to register unchanged. Manual tracing can do the same with tracer.trace(..., metadata={"tools": my_tools_array}). Click Register, review the draft, save - and because the failure history was keyed by the tool’s name all along, the accumulated evidence feeds Suggest improvement immediately.

Price per call

Some tools are not free to run: a geocoding API, a search provider, a KYC check, a paid MCP server. A registered tool can carry a price per call in USD, set when you register it or any time later from its detail dialog (the Price / call column shows it in the registry). From then on every recorded call of that tool is priced, and the spend shows up everywhere LLM spend does:
  • the Overview Total cost chart gets a Tool calls segment, with a per-tool breakdown under it in the legend (production traffic; tool calls made during eval runs are filed under the chart’s Eval runs segment, since that is evaluation spend whatever bills it),
  • the Monitor tab’s cost card gets a Tool calls series in both its views,
  • the alert rules estimatedCostUsd metric counts it, so a spend alert fires on a tool-call surge the same way it fires on a token surge.
Two rules keep the number honest. Every recorded call is charged, failed ones included: an external API bills the request, not the outcome, and a retry storm against a failing endpoint is exactly the spend you want to see. And an unpriced tool costs nothing in the chart, the same posture as a model with no pricing; the price is yours to maintain, from your provider’s invoice, not something the engine guesses. The price is matched on the traced tool-call name, the same join the failure evidence uses, so a tool traced as geocode must be registered as geocode. Pricing is metadata: changing it never creates a definition version. From the SDK:
On the wire, pricePerCallUsd is accepted by POST /evaluate/tool-schemas and PATCH /evaluate/tool-schemas/:id (a number of USD, or null to clear; negative or non-numeric values are refused with a 400), and read back on every tool schema. The cost chart’s GET /api/v1/agent-monitoring/cost-trend response carries the segment under the reserved key "tool calls" in totalsByModel / points[].byModel, plus totalsByTool for the per-tool split.

Test endpoint

A registered tool can optionally carry a test endpoint - an http(s) URL the Playground’s “From Tools & MCPs” picker uses as the tool’s endpoint default, so a real-loop test of a registered tool needs no re-typing. Calls POST {tool, arguments} and expect {result} back. It is only ever called from a Playground run or a conversation simulation you start: the registry itself stays execution-free, and production traffic always runs your agent’s own real tool. Set or clear it any time via PATCH /evaluate/tool-schemas/:id/test-endpoint, or via PATCH /evaluate/tool-schemas/:id, which updates the tool’s description, testEndpointUrl, and pricePerCallUsd together - metadata edits only, never the versioned definition.

MCP tools

Tools served by a remote MCP server register into this same registry - see MCPs for connecting servers (including OAuth-protected ones) and how Playground runs execute MCP tools over the real protocol.