TypeScript CI SDK

@agentx/eval is a zero-dependency TypeScript client for the CI slice of offline evaluation: open a run against an existing dataset, submit your agent’s outputs (the engine judge-scores each batch synchronously), finalize, and gate the build. Node 18+, ESM and CJS, nothing else - it uses the built-in fetch. Be clear about what it is: the CI surface only. Dataset curation, grading configs, the review queue, analysis reports, and everything else on these pages live in the Python SDK and the dashboard. Use this package when the repo being gated is TypeScript and pulling in a Python step just for the gate is the only reason it would exist.

yarn add @agentx/eval

import { AgentXEval } from "@agentx/eval";

const evals = new AgentXEval({
  apiKey: process.env.AGENTX_API_KEY!,
  baseUrl: "http://localhost:4700/api/v1",
});

const run = await evals.initRun({
  datasetId: "existing-dataset-id",
  subject: { name: "support-agent", metadata: { version: process.env.GIT_SHA } },
});

const outputs = await runMyAgentOverCases(); // your code
await run.submit(
  outputs.map((o, caseIndex) => ({
    caseIndex,
    query: o.query,
    output: o.answer,
    traceId: o.traceId, // optional: links the trace for trajectory-aware judging
  }))
);

const summary = await run.finalize();
console.log(summary.liveStatistics); // averageRating, ratedCount, skippedCount, failedCount

const gate = await run.gate({ failUnder: 7, noRegression: true, record: true, caller: "ci" });
gate.assert(); // throws an Error naming each failed check when the gate did not pass

The gate checks are the same ones the Python gate runs - failUnder (absolute floor) and noRegression (tolerance default 0.5, needs run history) - and recorded gates land in the same dashboard CI Gates history. Two more pieces round out the CI story. Pairwise comparison against a previous run:

const cmp = await evals.comparePairwise({ runAId: run.runId, runBId: "previous-run-id", bothOrders: true });
console.log(cmp.summary); // { total, aWins, bWins, ties, winner, flipRate }

And resume after an interrupted job - submittedKeys() returns what the engine already has, so a re-run only submits the remainder:

const done = new Set(await run.submittedKeys());
const remaining = allCases.filter((c, i) => !done.has(run.idempotencyKey(i)));

submit() auto-chunks at 10 results per batch and retries each batch once; a second failure throws an AgentXEvalError with the HTTP status rather than finishing silently. The package README in the AgentX-Trace-Eval repo (packages/agentx-eval) is the full reference.

Python SDK

TypeScript SDK

Integrations

TypeScript CI SDK