TypeScript CI SDK

@agentx/eval is a zero-dependency TypeScript client for the CI slice of offline evaluation: open a run against an existing dataset, submit your agent’s outputs (the engine judge-scores each batch synchronously), finalize, and gate the build. Node 18+, ESM and CJS, nothing else - it uses the built-in fetch. Be clear about what it is: the CI surface only. Grading configs, the review queue, analysis reports, and everything else on these pages live in the Python SDK and the dashboard - though the package does include a minimal evals.createDataset({ name, evaluationCriteria, questions }) for bootstrapping a dataset from the pipeline itself. Use this package when the repo being gated is TypeScript and pulling in a Python step just for the gate is the only reason it would exist.

yarn add @agentx/eval

The package is not yet on the public npm registry - build it from AgentX-trace-eval/packages/agentx-eval and install via a file: path or your private registry until it is published.

import { AgentXEval } from "@agentx/eval";

const evals = new AgentXEval({
  apiKey: process.env.AGENTX_API_KEY!,
  baseUrl: "http://localhost:4700/api/v1",
});

const run = await evals.initRun({
  datasetId: "existing-dataset-id",
  subject: { name: "support-agent", metadata: { version: process.env.GIT_SHA } },
});

const outputs = await runMyAgentOverCases(); // your code
await run.submit(
  outputs.map((o, caseIndex) => ({
    caseIndex,
    query: o.query,
    output: o.answer,
    traceId: o.traceId, // optional: links the trace for trajectory-aware judging
  }))
);

const summary = await run.finalize();
console.log(summary.liveStatistics); // averageRating, ratedCount, skippedCount, failedCount

const gate = await run.gate({ failUnder: 7, noRegression: true, record: true, caller: "ci" });
gate.assert(); // throws an Error naming each failed check when the gate did not pass

The gate checks are the same ones the self-host CI gate runs - failUnder (absolute floor) and noRegression (tolerance default 0.5, needs run history) - and recorded gates land in the same dashboard CI Gates history. For noRegression, the baseline must match the run’s evaluationSettingsId, scorerGroupId, and split, trace evaluations are excluded, only the 5 newest candidates are considered, and with no matching run the check passes explicitly. Two more pieces round out the CI story. Pairwise comparison against a previous run:

const cmp = await evals.comparePairwise({ runAId: run.runId, runBId: "previous-run-id", bothOrders: true });
console.log(cmp.summary); // { total, aWins, bWins, ties, winner, flipRate }

And resume after an interrupted job - submittedKeys() returns what the engine already has, so a re-run only submits the remainder:

const done = new Set(await run.submittedKeys());
const remaining = allCases.filter((c, i) => !done.has(run.idempotencyKey(i)));

submit() auto-chunks at 10 results per batch and retries each batch once; a second failure throws an AgentXEvalError with the HTTP status rather than finishing silently. The package README in the AgentX-trace-eval repo (packages/agentx-eval) is the full reference.

Self-Host CI Gate

Hosted CI/CD

TypeScript CI SDK