Playground

Before committing to a full dataset run, test a prompt against real models and real test cases interactively - modeled on Braintrust’s and OpenAI’s own playgrounds. Governance → Playground, its own top-level tab:

• Edit a multi-turn prompt - system message, optional few-shot user/assistant example turns - freely, or load a saved prompt’s current version from the Prompt Registry as a starting point.
• Select one or more OpenAI, Anthropic, and/or Gemini models, from the same dashboard-managed model catalog Model portability uses.
• Pick an existing dataset and check which of its questions to include.
• Run (the button reads “Run N cells” once models and cases are picked) fires the whole (question × model) grid for real - each cell independently shows the actual output, latency, tokens, and estimated cost, plus a judge rating whenever that question has an expected answer, and each of the dataset’s enabled code scorers’ results alongside it. At most 4 cells run at once (a fixed concurrency cap, so a large grid doesn’t trip your provider’s rate limits) - the rest queue and fill in as earlier ones finish.

Playground is a scratchpad with a memory: nothing here creates a real eval run or saves a prompt version on its own, but past runs are kept in History (the breadcrumb’s Edit | History toggle, with a Runs | Simulations switch inside) so a refresh doesn’t lose your grid, and past simulations reopen with their full transcript. Copy a result you like into a saved Prompt or a real dataset run once you’re happy with it.

​

Scorers: one panel, two roles

The sidebar’s single Scorers panel covers both grading and monitoring - every LLM judge scorer and custom pattern is one row:

• Exactly one judge carries the star: it grades the run - reference-based against the dataset case’s expected results, supplying the score, similarity metrics, and repetitions (the workspace’s default scorer is preselected, provisioned automatically on first use).
• Checked rows additionally dry-run the way live monitoring would: each cell gets a badge per check - a pattern’s reads “Matched” / “No match”, a judge’s shows its reference-free rating. Nothing here saves signals or spends beyond the checks you tick. Starring a judge removes it from the check set - the grader never runs twice.
• Row hover Edit and New judge scorer open the unified scorer editor without leaving the Playground; a freshly created judge becomes the grader.

A single query (no dataset case) has no expected answer, so the grader is skipped and checked judges still score reference-free.

​

Tools

The Playground tests prompt and tools together. Add a tool ad hoc, or pull one straight from the Tools & MCPs registry with the From Tools & MCPs picker - name, description, and parameter schema come in as the registry’s current published version. The endpoint URL is optional:

• Blank (simulated): the tool is still sent to the model, and its calls return a canned {simulated: true} result - so whether the prompt/model choose the tool and form valid arguments is fully testable with zero infrastructure. The cell’s tool-call trace shows the arguments and marks the result simulated, so nothing pretends to be a real lookup.
• Set: each call POSTs {tool, arguments} to your real service and feeds the actual response back to the model - the full loop, including how the prompt handles real tool outputs.

​

Beyond single answers

The grid tests single-shot answers. Simulate conversation tests multi-turn behavior with a persona-driven simulated user, and Model portability replays any captured trace against alternative models for a cost/latency/quality comparison.