How the core objects relate: traces group into sessions; monitors raise signals; datasets drive eval runs; findings become registry proposals; auto improvements
Tracing
| Term | Meaning |
|---|---|
| Trace | One agent interaction: a root record with input, output, latency, model, tokens, and tool calls. |
| Span | One step inside a trace - a graph node, LLM call, tool execution, or retrieval - linked by span_id/parent_span_id into a tree. The trace dialog’s Timeline and Graph views render this tree. |
| Session | The conversation a trace belongs to: every trace sharing a session_id. Sessions get their own view, turn counts, and conversation-level judging. See Sessions. |
| Tool call | A recorded tool invocation (name, arguments, result, success flag). Feeds tool-quality metrics, the Tool-failure check, and trajectory matching. |
| Agent | The named producer of traces. Auto-registered on first ingest; carries a monitoring profile and health rate in the Agents tab. |
Monitoring
| Term | Meaning |
|---|---|
| Scorer | Anything that can pass judgment on traffic, managed from the one Scorers page: template/pattern rules, LLM judges, and custom endpoints. All opt-in - nothing is judged until a scorer is enabled. |
| Pattern | A deterministic scorer: one of the six shipped templates (secrets, PII, prompt-injection echo, profanity, refusal, malformed JSON) or a rule you author (keyword, regex, or LLM-judged rubric rows). Polarity is failure (raises an issue) or proper (logs a healthy tally). |
| Operational outcome | NOT a scorer: a fact the trace itself recorded - an error, a failed tool call, an empty response. Always classified into the KPI failure metrics, never raises a triage signal, nothing to enable. |
| Signal | One deduped finding: repeated matches of the same scorer for the same agent accumulate on one row (occurrence count, last-seen) instead of flooding the list. Triage with statuses, archive what’s done - a re-firing archived signal reopens automatically. |
| LLM judge scorer | An LLM judge that scores live traffic continuously against an evaluator config’s criteria - per trace at ingest, or per session once the conversation goes idle (scope="session"). Low scores raise signals. (“Online evaluator” in the SDK/API.) |
| Custom scorer | Your own logic per sampled trace: an external HTTP endpoint returning a verdict, or a code scorer - a Python/JavaScript handler() you write, run in-engine, scoring 0-1. (“Custom evaluator” in the SDK/API.) |
| User feedback | An end user’s thumbs up/down forwarded from your app - human ground truth attached to the trace: raises a triage signal on a downvote, drives the Downvote rate KPI, and calibrates the judges. Not a scorer; nothing to configure. |
| Topic | Automatic clustering of traffic by subject, shown on the Overview topic map. |
| Outcome | A real-world result you report back (client.outcomes.report) - refund issued, ticket reopened - used by Judge Calibration to compare judge verdicts against reality. |
Evaluation
| Term | Meaning |
|---|---|
| Dataset | A versioned, reusable set of test cases plus scoring configuration (criteria, metrics, code scorers, CI settings). Runs snapshot the dataset version they used. |
| Question | One test case: query, optional expectedResults (the reference answer), optional per-case judgeGuideline, optional expected trajectory, optional smoke-test variants (LLM-paraphrased rewordings to catch phrasing brittleness). |
| Evaluator config | A standalone grading configuration (criteria + judge prompt + judge model) reusable across datasets and online evaluators. |
| Evaluation run | One execution of a dataset against an agent: every output, its scores, the locked dataset version, and the subject description. Statuses: in_progress → completed or failed. |
| Judge rating | The LLM judge’s 0-10 score with a written justification. When a result links its trace, the judge also sees the execution trajectory - tools called, order, failures - not just the final text. |
| Similarity metrics | Optional per-dataset scorers against expectedResults: vector (cosine) similarity, Jaccard token overlap, BLEU, ROUGE-L. |
| Code scorer | A sandboxed JavaScript function scoring each result - deterministic checks (format, keywords, tool behavior) that need no LLM. |
| Trajectory match | A deterministic pass/fail comparing the tool calls a linked trace actually made against the case’s expected_tools, in one of four modes: strict (same calls, same order), unordered (same calls, any order), superset (all expected present, extras allowed), subset (nothing unexpected, missing allowed). |
| Evaluation subject | Metadata describing what was evaluated (kind, displayName, framework, optional version for version comparison). |
CI/CD
| Term | Meaning |
|---|---|
| Gate | The pass/fail decision for a pipeline: rating floor (fail_under) and/or no-regression versus a baseline run. |
| Pass rate | Fraction of cases passing every threshold rule (0.0-1.0). |
| Threshold | A per-metric rule on the dataset (“judge rating ≥ 7”, “cosine ≥ 0.8”) counted into the pass rate. |
| Fail fast | Finalize the run as failed on the first threshold violation instead of waiting for remaining cases. |
Improve
| Term | Meaning |
|---|---|
| Prompt registry | Versioned prompts your agent pulls at runtime (client.evaluations.prompts), with judge-proposed, human-approved rewrites. |
| Tool schema registry | Versioned tool definitions with failure evidence gathered from traces and playground runs, and “Suggest improvement” proposals. |
| Proposal validation | Before adopting a proposed change, AgentX runs candidate-vs-current against a dataset and shows the measured difference. |