tools=[...]) are the other half, and a vague parameter description causes malformed tool calls the same way a vague prompt causes bad answers. Tools & MCPs in the sidebar’s Manage section gives them the same registry treatment as prompts: version-scoped, edited in a schema-aware JSON editor, with full version history.
Suggest improvement works the same way as a prompt’s, fed by tool-specific evidence: recorded tool calls that failed (success: false, the same field tracer.trace_tool_call captures automatically when your tool raises) and low-rated online-evaluator traces that used the tool. The judge proposes a rewritten definition - typically tightening parameter descriptions and enums against the failures it saw - shown as an editable diff, and nothing publishes until you approve it. Your agent’s code then pulls the published definition instead of hardcoding it, the same source-of-truth pattern as the prompt registry.
The Playground can also add a tool straight from this registry, so you can test a proposed definition against real models before publishing it.
The whole loop is scriptable via client.evaluations.tool_schemas - the tool-definition
analog of the prompt registry’s SDK path:
Tools you haven’t registered yet
Failure classification was never registry-gated:agent-tool-failure:<name> events tally for any traced tool (an operational outcome, not a scorer - it feeds the KPI metrics and this evidence loop directly, without raising triage signals). What registration unlocks is the improvement loop - so the Tool Schemas view has a Registered / Unregistered toggle, with live counts on each side. Unregistered lists tool names observed in recent traces that aren’t in the registry yet, with call/failure counts and a drafted definition ready to review - verbatim from the trace’s metadata.tools when available, otherwise inferred from the arguments the model actually formed (marked as inferred).
The SDK’s raw-client integrations (patch_openai_client, patch_anthropic_client, the LiteLLM logger, Google GenAI) and the LangChain/LangGraph handler capture the request’s tools=[...] definitions automatically and attach them as metadata.tools - so any tool your agent actually passed to the model surfaces here with its real name, description, and parameter schema, ready to register unchanged. Manual tracing can do the same with tracer.trace(..., metadata={"tools": my_tools_array}). Click Register, review the draft, save - and because the failure history was keyed by the tool’s name all along, the accumulated evidence feeds Suggest improvement immediately.
Price per call
Some tools are not free to run: a geocoding API, a search provider, a KYC check, a paid MCP server. A registered tool can carry a price per call in USD, set when you register it or any time later from its detail dialog (the Price / call column shows it in the registry). From then on every recorded call of that tool is priced, and the spend shows up everywhere LLM spend does:- the Overview Total cost chart gets a Tool calls segment, with a per-tool breakdown under it in the legend (production traffic; tool calls made during eval runs are filed under the chart’s Eval runs segment, since that is evaluation spend whatever bills it),
- the Monitor tab’s cost card gets a Tool calls series in both its views,
- the alert rules
estimatedCostUsdmetric counts it, so a spend alert fires on a tool-call surge the same way it fires on a token surge.
geocode must be registered as geocode. Pricing is metadata: changing it
never creates a definition version.
From the SDK:
pricePerCallUsd is accepted by POST /evaluate/tool-schemas and
PATCH /evaluate/tool-schemas/:id (a number of USD, or null to clear; negative or
non-numeric values are refused with a 400), and read back on every tool schema. The cost chart’s
GET /api/v1/agent-monitoring/cost-trend response carries the segment under the reserved key
"tool calls" in totalsByModel / points[].byModel, plus totalsByTool for the per-tool
split.
Test endpoint
A registered tool can optionally carry a test endpoint - an http(s) URL the Playground’s “From Tools & MCPs” picker uses as the tool’s endpoint default, so a real-loop test of a registered tool needs no re-typing. Calls POST{tool, arguments} and expect {result} back. It is only ever called from a Playground run or a conversation simulation you start: the registry itself stays execution-free, and production traffic always runs your agent’s own real tool. Set or clear it any time via PATCH /evaluate/tool-schemas/:id/test-endpoint, or via PATCH /evaluate/tool-schemas/:id, which updates the tool’s description, testEndpointUrl, and pricePerCallUsd together - metadata edits only, never the versioned definition.

