> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# TypeScript CI SDK

> @agentx/eval - the minimum eval slice for a TypeScript pipeline: submit, finalize, gate

`@agentx/eval` is a zero-dependency TypeScript client for the CI slice of offline evaluation:
open a run against an existing dataset, submit your agent's outputs (the engine judge-scores
each batch synchronously), finalize, and gate the build. Node 18+, ESM and CJS, nothing else -
it uses the built-in `fetch`.

Be clear about what it is: **the CI surface only**. Grading configs, the review queue,
analysis reports, and everything else on these pages live in the
[Python SDK](/sdk/evaluations/overview) and the dashboard - though the package does include a
minimal `evals.createDataset({ name, evaluationCriteria, questions })` for bootstrapping a
dataset from the pipeline itself. Use this package when the repo being gated is TypeScript and
pulling in a Python step just for the gate is the only reason it would exist.

```bash theme={null}
yarn add @agentx/eval
```

<Note>
  The package is not yet on the public npm registry - build it from
  `AgentX-trace-eval/packages/agentx-eval` and install via a `file:` path or your private
  registry until it is published.
</Note>

```ts theme={null}
import { AgentXEval } from "@agentx/eval";

const evals = new AgentXEval({
  apiKey: process.env.AGENTX_API_KEY!,
  baseUrl: "http://localhost:4700/api/v1",
});

const run = await evals.initRun({
  datasetId: "existing-dataset-id",
  subject: { name: "support-agent", metadata: { version: process.env.GIT_SHA } },
});

const outputs = await runMyAgentOverCases(); // your code
await run.submit(
  outputs.map((o, caseIndex) => ({
    caseIndex,
    query: o.query,
    output: o.answer,
    traceId: o.traceId, // optional: links the trace for trajectory-aware judging
  }))
);

const summary = await run.finalize();
console.log(summary.liveStatistics); // averageRating, ratedCount, skippedCount, failedCount

const gate = await run.gate({ failUnder: 7, noRegression: true, record: true, caller: "ci" });
gate.assert(); // throws an Error naming each failed check when the gate did not pass
```

The gate checks are the same ones the [self-host CI gate](/integrations/self-host-ci) runs -
`failUnder` (absolute floor) and `noRegression` (tolerance default 0.5, needs run history) -
and recorded gates land in the same dashboard CI Gates history. For `noRegression`, the
baseline must match the run's `evaluationSettingsId`, `scorerGroupId`, and split, trace
evaluations are excluded, only the 5 newest candidates are considered, and with no matching
run the check passes explicitly.

Two more pieces round out the CI story. Pairwise comparison against a previous run:

```ts theme={null}
const cmp = await evals.comparePairwise({ runAId: run.runId, runBId: "previous-run-id", bothOrders: true });
console.log(cmp.summary); // { total, aWins, bWins, ties, winner, flipRate }
```

And resume after an interrupted job - `submittedKeys()` returns what the engine already has,
so a re-run only submits the remainder:

```ts theme={null}
const done = new Set(await run.submittedKeys());
const remaining = allCases.filter((c, i) => !done.has(run.idempotencyKey(i)));
```

`submit()` auto-chunks at 10 results per batch and retries each batch once; a second failure
throws an `AgentXEvalError` with the HTTP status rather than finishing silently. The package
README in the `AgentX-trace-eval` repo (`packages/agentx-eval`) is the full reference.