Get Run

curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE" \
  -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"

run = client.evaluations.get_run("rK7dP2qWx9TzB4mV6nJcE")
print(run["status"], run["averageRating"])

{
  "_id": "rK7dP2qWx9TzB4mV6nJcE",
  "runId": "rK7dP2qWx9TzB4mV6nJcE",
  "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
  "evaluationSettingsId": null,
  "scorerGroupId": null,
  "scorerGroupName": null,
  "additionalScorerIds": null,
  "scorerBreakdown": [
    { "scorerId": null, "name": "Primary scorer", "primary": true, "averageRating": 8, "scored": 1 }
  ],
  "evaluationSubject": {
    "kind": "custom_agent",
    "displayName": "Customer Support Bot",
    "framework": "langchain"
  },
  "status": "completed",
  "resultCount": 1,
  "averageRating": 8,
  "liveStatistics": {
    "averageRating": 8,
    "minRating": 8,
    "maxRating": 8,
    "ratedCount": 1,
    "skippedCount": 0,
    "failedCount": 0
  },
  "caseStatistics": [],
  "results": [
    {
      "questionIndex": 0,
      "questionText": "How do I reset my password?",
      "runNumber": 1,
      "input": { "query": "How do I reset my password?" },
      "output": { "text": "Click Forgot Password on the login screen." },
      "rating": 8,
      "justification": "Accurate answer, empathetic tone.",
      "traceId": "mJ3vQ8pTr2LqYw6bZk9Xd",
      "latencyMs": 1240,
      "inputTokens": 150,
      "outputTokens": 45,
      "vectorSimilarity": null,
      "jaccardSimilarity": 0.62,
      "bleuScore": null,
      "rougeScore": null,
      "codeScorerResults": null,
      "judgeScorerResults": null,
      "isSmokeTestVariant": false,
      "smokeTestVariantText": null,
      "status": "scored",
      "error": null
    }
  ]
}

{
  "error": "Run not found"
}

GET

api

custom-agent-evaluations

runs

{runId}

curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE" \
  -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"

run = client.evaluations.get_run("rK7dP2qWx9TzB4mV6nJcE")
print(run["status"], run["averageRating"])

{
  "_id": "rK7dP2qWx9TzB4mV6nJcE",
  "runId": "rK7dP2qWx9TzB4mV6nJcE",
  "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
  "evaluationSettingsId": null,
  "scorerGroupId": null,
  "scorerGroupName": null,
  "additionalScorerIds": null,
  "scorerBreakdown": [
    { "scorerId": null, "name": "Primary scorer", "primary": true, "averageRating": 8, "scored": 1 }
  ],
  "evaluationSubject": {
    "kind": "custom_agent",
    "displayName": "Customer Support Bot",
    "framework": "langchain"
  },
  "status": "completed",
  "resultCount": 1,
  "averageRating": 8,
  "liveStatistics": {
    "averageRating": 8,
    "minRating": 8,
    "maxRating": 8,
    "ratedCount": 1,
    "skippedCount": 0,
    "failedCount": 0
  },
  "caseStatistics": [],
  "results": [
    {
      "questionIndex": 0,
      "questionText": "How do I reset my password?",
      "runNumber": 1,
      "input": { "query": "How do I reset my password?" },
      "output": { "text": "Click Forgot Password on the login screen." },
      "rating": 8,
      "justification": "Accurate answer, empathetic tone.",
      "traceId": "mJ3vQ8pTr2LqYw6bZk9Xd",
      "latencyMs": 1240,
      "inputTokens": 150,
      "outputTokens": 45,
      "vectorSimilarity": null,
      "jaccardSimilarity": 0.62,
      "bleuScore": null,
      "rougeScore": null,
      "codeScorerResults": null,
      "judgeScorerResults": null,
      "isSmokeTestVariant": false,
      "smokeTestVariantText": null,
      "status": "scored",
      "error": null
    }
  ]
}

{
  "error": "Run not found"
}

Returns one run: its status, rating statistics, per-case repetition spread, and every submitted result with its scores. Poll this while a run is "in_progress", or read it back after finalizing.

Authentication

string

required

Project API key.

Path Parameters

string

required

Run ID returned by POST /runs.

Response

Field	Type	Description
`_id`	string	Run ID
`runId`	string	Same value as `_id` (historical key, kept for SDK compatibility)
`datasetId`	string	Dataset the run scores against
`evaluationSettingsId`	string \| null	Standalone grading config, if one was passed to `POST /runs`. Reads back `null` when a `scorerGroupId` was also passed - the group supersedes it (`additionalScorerIds` likewise reads back `null` in that case)
`scorerGroupId`	string \| null	Scorer group grading this run, if one was passed to `POST /runs` (Scorer groups)
`scorerGroupName`	string \| null	Resolved name of that scorer group; `null` when the group was deleted. With `scorerGroupId` still set, that is how a client detects the deletion
`additionalScorerIds`	string[] \| null	Extra judge scorer ids passed to `POST /runs`, if any
`scorerBreakdown`	array	Per-scorer aggregate, always present, primary row first: `[{ scorerId, name, primary, averageRating, scored }]`. The primary row’s `scorerId` is the grading config or scorer group id (`null` when the run graded on the dataset’s own rubric), and it is named `"<group> (group)"` for scorer-group runs, else the grading config’s name, else `"Primary scorer"`
`evaluationSubject`	object \| null	The subject metadata passed to `POST /runs` (including any `split` tag)
`status`	string	`"in_progress"` \| `"completed"` \| `"failed"`
`resultCount`	number	Total submitted results (all statuses, smoke-test variants included)
`averageRating`	number \| null	Mean judge rating across rated results (smoke-test variants excluded, as in `liveStatistics` and `scorerBreakdown`)
`liveStatistics`	object	`{ averageRating, minRating, maxRating, ratedCount, skippedCount, failedCount }`. `skippedCount` = results the judge could not score (e.g. missing judge key or reference); `failedCount` = results submitted with an `error`
`caseStatistics`	array	Per-case repetition spread, for cases with 2+ rated rows (smoke-test variants excluded): `[{ questionIndex, ratedCount, averageRating, minRating, maxRating, ratingVariance }]`
`results`	array	Per-result rows, see below

array

Each result row:

Field	Type	Description
`questionIndex`	number \| null	0-based dataset question index
`questionText`	string \| null	The question’s `query`, resolved from the dataset
`runNumber`	number \| null	Repetition number (1 through `numberOfRequests`)
`input`	object \| null	As submitted, e.g. `{ "query": "..." }`
`output`	object \| null	As submitted, e.g. `{ "text": "..." }`
`rating`	number \| null	Judge score 0-10; `null` when skipped/failed
`justification`	string \| null	Judge explanation, or the skip reason
`traceId`	string \| null	Linked trace, if the result carried one
`latencyMs` / `inputTokens` / `outputTokens`	number \| null	From the submitted `timings`
`vectorSimilarity` / `jaccardSimilarity` / `bleuScore` / `rougeScore`	number \| null	Similarity scores, when the metric is enabled and the case has `expectedResults`
`codeScorerResults`	array \| null	Code scorer, trajectory match, and context match rows: `[{ name, score, reasoning?, error? }]`
`judgeScorerResults`	array \| null	One verdict per additional judge scorer: `[{ scorerId, name, rating, justification }]` (`rating: null` = that judge could not score the row). The primary scorer’s verdict stays in `rating`/`justification`
`isSmokeTestVariant`	boolean	Whether this row is a paraphrase variant
`smokeTestVariantText`	string \| null	The variant phrasing, if so
`status`	string	`"scored"`, `"skipped"` (judge could not score), or `"failed"` (result carried an `error`)
`error`	object \| null	The submitted `{ type, message }` error, if any

Errors

Status	Body	Meaning
`404`	`{ "error": "Run not found" }`	No run with this id in the project

curl "http://localhost:4700/api/v1/custom-agent-evaluations/runs/rK7dP2qWx9TzB4mV6nJcE" \
  -H "x-api-key: agtx_local_0f3c9a17d2b84e6a5c01b9f4e7d8a2c6431b5f97a0e2d4c8"

run = client.evaluations.get_run("rK7dP2qWx9TzB4mV6nJcE")
print(run["status"], run["averageRating"])

{
  "_id": "rK7dP2qWx9TzB4mV6nJcE",
  "runId": "rK7dP2qWx9TzB4mV6nJcE",
  "datasetId": "dS4tG7hNb2VxZ8kQ5wMyA",
  "evaluationSettingsId": null,
  "scorerGroupId": null,
  "scorerGroupName": null,
  "additionalScorerIds": null,
  "scorerBreakdown": [
    { "scorerId": null, "name": "Primary scorer", "primary": true, "averageRating": 8, "scored": 1 }
  ],
  "evaluationSubject": {
    "kind": "custom_agent",
    "displayName": "Customer Support Bot",
    "framework": "langchain"
  },
  "status": "completed",
  "resultCount": 1,
  "averageRating": 8,
  "liveStatistics": {
    "averageRating": 8,
    "minRating": 8,
    "maxRating": 8,
    "ratedCount": 1,
    "skippedCount": 0,
    "failedCount": 0
  },
  "caseStatistics": [],
  "results": [
    {
      "questionIndex": 0,
      "questionText": "How do I reset my password?",
      "runNumber": 1,
      "input": { "query": "How do I reset my password?" },
      "output": { "text": "Click Forgot Password on the login screen." },
      "rating": 8,
      "justification": "Accurate answer, empathetic tone.",
      "traceId": "mJ3vQ8pTr2LqYw6bZk9Xd",
      "latencyMs": 1240,
      "inputTokens": 150,
      "outputTokens": 45,
      "vectorSimilarity": null,
      "jaccardSimilarity": 0.62,
      "bleuScore": null,
      "rougeScore": null,
      "codeScorerResults": null,
      "judgeScorerResults": null,
      "isSmokeTestVariant": false,
      "smokeTestVariantText": null,
      "status": "scored",
      "error": null
    }
  ]
}

{
  "error": "Run not found"
}

Tracing

Custom Evaluations

CI Gate

Agent Registry

Authentication

Path Parameters

Response

Errors

​Authentication

​Path Parameters

​Response

​Errors

Authentication

Path Parameters

Response

Errors