Skip to main content
A pattern is a detection rule checked against your traffic: when a trace matches, a signal is raised (or a healthy tally recorded) in the same triage queue everything else feeds. Patterns are the deterministic kind on the Scorers page - cheap and exact - alongside the six shipped built-in templates, while LLM judge scorers are the judgment kind, scoring quality continuously with a real LLM. Use a pattern when you can say precisely what bad looks like (“promises a refund”, “mentions a competitor”, “apologizes more than once”); use an LLM judge when you can only describe it.

​
Building a pattern

Scorers → New scorer → Pattern scorer. A pattern is one or more condition rows, evaluated top to bottom:
  • Detector - each row is one of three kinds:
    • Phrase: plain-text contains match (case sensitivity is a per-row toggle).
    • Regex: a regular expression body. Don’t want to write one? Generate from description drafts the regex from plain English with an LLM call, right in the dialog.
    • Semantic: a rubric an LLM judges the text against (“The response promises a refund.”). The only detector kind that needs a provider key configured.
  • Match target - where the row looks: response (the traced output), userMessage (the traced input), or trace (output plus error plus every recorded tool call’s name/input/output, flattened - the right target for “did any tool mention X”).
  • Negate - flips a single row’s verdict before it joins the others.
  • Connector - rows combine top to bottom with AND, OR, or NOR (joins as “and not”), so “contains ‘refund’ AND NOT matches the approved-refund-template regex” is two rows.

​
Behavior settings

Phrase and regex detection work with no provider keys at all - only semantic rows need one.

​
Where matches go

Matches are deduped by pattern and agent: a recurring issue accumulates an occurrence count on one signal instead of a new row per trace. From a signal you can triage its status, open the exact traces that matched, draft human feedback with an LLM assist, or create an evaluator from the signal - turning a recurring rule match into a continuous quality score. Finished with a signal? Archive it (per row, or “Archive selected” in bulk) - archived signals leave every filter except the explicit Archived shelf, so the queue never grows unbounded. Archiving is a shelf, not a grave: if the same issue fires again, the signal reopens into the active list automatically.

​
From the SDK

Patterns are first-class SDK resources: client.monitor.patterns.builder(...).publish() returns an id you can pass at trace time via pattern_ids. The Monitor page documents the builder’s full parameter table, the built-in checks that run alongside your custom patterns, and reading signals back with client.monitor.signals.

Online Evaluators

The judgment half: continuous LLM scoring of live traffic

Custom Evaluators

Delegate the verdict to your own HTTP endpoint