Skip to main content
Your datasets say what you test. Production says what actually happens. Dataset coverage (Insights → Dataset coverage) is the join: it groups classified production traffic into topics, measures how deeply your dataset cases cover each topic, and ranks the gaps by what closing them is worth.

​
Prerequisites

  • Topics classification on (Settings → Monitoring Defaults) - coverage is measured against the topics Monitor derives from real traffic, so an install with nothing classified shows an empty state, not zeroes.
  • OPENAI_API_KEY for embeddings. Without one the page still works but runs in a labelled Approximate mode: coverage falls back to counting cases against a per-topic target instead of measuring depth, and off-map detection is disabled (lexical matching cannot prove a case matches nothing).

​
The three headline numbers

A fourth card, Cases off the map, lists dataset cases matching no topic production asks - candidates to retire (each opens with its expected results and nearest topic, so retiring is a read decision, not a guess).

​
Coverage is depth, not case count

A topic’s coverage is a facility-location value over the topic’s real traces: each production trace is scored against its nearest case, so near-duplicate cases add nothing and the number cannot be inflated by generating copies. A topic whose traces carry no embeddings falls back to case-count-against-target for that topic alone, and says so (“cases against target - no depth available”). The “covered” threshold is the same similarity band addCaseToDataset uses for dedupe - covered literally means the dataset would reject this query as a duplicate.

​
Working the queue

The topics table ranks by priority = traffic exposed x (1 - coverage) x risk amplification - a covered topic is worth nothing to work on however busy, and a risky topic outranks a healthy one carrying several times its traffic. The Next up chips surface the top three. Selecting a topic opens its detail rail:
  • the four per-topic numbers (traffic with trace/session counts, cases against target, coverage with its basis, observed risk with its issue/negative split),
  • Production traces in this topic - the newest real requests with the classifier’s verdict, so a gap is judged by reading what production actually asked. Each trace in the list carries an Add test case button, the manual path for turning that one request into a case,
  • Suggested action with one button: Generate N cases from traces creates candidate cases from the topic’s own traces through the standard curation path (deduped, versioned, expectedResults deliberately left for you to write - a generated case is a scaffold to review, never a fabricated assertion).

​
The probe: “is this already tested?”

The probe card answers coverage for one specific query - or a pasted list (a launch spec, a support macro export) as a pre-launch gate. Verdicts distinguish a real gap (production asks this, nothing tests it) from untested and unasked (nothing tests it, but nobody asks - deliberately neutral, because writing that test is work nobody needs).

​
The coverage map

The Map tab draws production traffic and your dataset cases in one picture: a joint UMAP projection in question space - a trace’s input-only embedding against a case’s query-only embedding - so an identical question from both sources lands in the same spot. (The coverage table’s topic matching runs in interaction space, input+output vs query+expected; that is the right space for “is this behavior tested” but the wrong one for this picture, since it separates identical questions whenever the actual answer differs from the expected one.) Blue dots are classified production traces; orange dots are dataset cases. A translucent density cloud glows around the points in each source’s color - clouds emerge only where points actually cluster, blue and orange blend where dataset cases sit on production traffic, and a blue region with no orange tint is the gap, visible without reading a number. Reading aids:
  • Source chips (Both / Production / Datasets) filter the view; the clouds follow.
  • The topic legend shows production · dataset point counts per topic - 34 · 0 is an untested topic before you have read the chart at all - and hovering a topic spotlights its points and clouds while dimming everything else.
  • Unmatched cases (tests no production topic claims) get a dashed chip; they are the same cases the Off-map tab lists.
  • The map inherits the page’s date range and dataset filter; cases or traces still warming show as an “N still indexing” chip rather than silently missing (traces classified before the question-space column existed backfill in bounded batches as the map is viewed).
Two honest limitations: UMAP coordinates carry no absolute meaning (only relative distance does, and the layout changes between fetches), and the map requires embeddings on both sides - unlike the coverage table it has no lexical fallback, because a position IS a similarity claim.

​
Scoping and export

The date range (presets from 6 hours to 90 days, or a custom range - custom spans are capped at one year; the default is the past 30 days) and the dataset filter scope every number on the page, the map included. Export writes exactly the rows on screen as CSV; the filename is prefixed provisional- when the numbers are still a floor (case embeddings indexing) or approximate- when they are lexical.

​
API

  • GET /api/v1/insights/coverage?window=30d&datasetId=... - the full coverage result.
  • GET /api/v1/insights/coverage/map - the coverage map (same scoping parameters; a separate route because the joint UMAP fit is heavier than the coverage sweep, so you only pay for it with the Map tab open). Needs at least 10 embedded points across both sources; below that it returns insufficientData: true with the reason.
  • POST /api/v1/insights/topics/curate with { topic, datasetId, window?, limit? } - the “Generate N cases from traces” action (returns { ok, added, duplicates, considered }). limit defaults to 6 and is capped at 12; anything larger is a bulk import, which has its own surface.
  • POST /api/v1/insights/probe and /probe/batch - the probe.
Both GET routes accept either window (24h / 7d / 30d) or an explicit custom range as from and to (epoch milliseconds, to > from, capped at one year) - the same range override every monitoring dashboard route takes. datasetId and datasetIds are interchangeable, each accepted repeated or comma-separated.