AI Analysis Report

.analyze() is the last step in the evaluation chain. It runs the same durable, multi-stage pipeline as the dashboard’s “Analyze” button: each response is scored by 1-3 LLM judges, then reduced through question- and cluster-level summaries into one final qualitative report, returned as a Report object.

report = client.evaluations.run(...).execute(my_agent).finalize().analyze(
    mode="auto",                                    # "auto" (default) | "sync" | "batch"
    quality_mode="quality_first",                   # "quality_first" (default) | "balanced"
    judges=["gpt-5.5", "claude-opus-4-8"],           # 1-3 model ids; omit for a single gpt-5.5 judge
)

report.summary                 # str | None, overall narrative summary
report.consistency_score       # float | None, 0-10, run-to-run consistency
report.instruction_adherence   # ReportInstructionAdherence | None
report.response_patterns       # ReportResponsePatterns | None
report.reasoning_analysis      # ReportReasoningAnalysis | None
report.tool_usage_analysis     # ReportToolUsageAnalysis | None
report.strengths               # list[str]
report.weaknesses              # list[str]
report.overall_rating          # str | None, "high" | "medium" | "low"
report.recommendations         # list[ReportRecommendation]

Numeric scores (average_rating, similarity metrics) are ready right after .finalize(), so you don’t need to wait for .analyze() just to see how the run went. Because .analyze() polls a job until it completes, it can take noticeably longer than a single LLM call for larger runs; progress is shown in the terminal while it waits.

Controlling the analysis

Parameter	Values	Description
`mode`	`"auto"` (default), `"sync"`, `"batch"`	How item scoring executes server-side; `"auto"` picks based on run size
`quality_mode`	`"quality_first"`, `"balanced"`	`"quality_first"` runs a second judge on every item; `"balanced"` samples based on risk
`judges`	1-3 model ids	Which LLM(s) score each response. The first always runs; a second confirms, a third only breaks a tie between the first two. Defaults to a single judge, `["gpt-5.5"]`, if omitted
`poll_interval`	seconds, default `5.0`	How often to check job status while waiting
`timeout`	seconds, default `1800.0`	Give up waiting after this long (the job keeps running server-side; call `get_report()` later to check on it)

Check on a long-running analysis without calling .analyze() again, even from a separate script execution:

status = client.evaluations.get_analysis_status(run_id)
print(status.status, status.progress.overall_percentage)

Fields

Field	Shape	Description
`summary`	`str`	Overall narrative summary of the run
`consistency_score`	`float`	0–10, how consistent responses were across repeated runs of the same question
`instruction_adherence`	`{ score, analysis, deviations: [str], rating }`	How well responses followed `subject.agentInstructions`
`response_patterns`	`{ similarities: [str], differences: [str], outliers: [str], rating }`	Cross-run consistency patterns
`reasoning_analysis`	`{ cot_quality, reasoning_patterns: [str], reasoning_gaps: [str], rating }`	Quality of the agent’s chain-of-thought, when traced
`tool_usage_analysis`	`{ effectiveness, patterns: [str], issues: [str], rating }`	How well the agent used its tools, when tool calls were traced
`strengths`	`list[str]`	What the agent did well
`weaknesses`	`list[str]`	Where the agent fell short
`overall_rating`	`"high" \| "medium" \| "low"`	Coarse overall quality rating
`recommendations`	`[{ category, priority, recommendation, reasoning }]`	Actionable, prioritized fixes

Each rating sub-field is one of "high", "medium", "low". recommendations[].category is one of "instructions", "tools", "knowledge", "reasoning", "consistency", "other"; priority is "high", "medium", or "low".

Reading it in your terminal

report = client.evaluations.run(...).execute(my_agent).finalize().analyze()

print(f"Rating: {report.overall_rating}")
print(report.summary)

for rec in report.recommendations:
    print(f"[{rec.priority}] {rec.category}: {rec.recommendation}")

Or open report.dashboard_url to review the full report, per-question breakdowns, and instruction-change suggestions in the dashboard.

Per-result rows without an analysis

results() returns the raw scored rows - no analysis job, no extra judge calls - for asserting on individual results in scripts and CI:

run = client.evaluations.run(...).execute(my_agent).finalize()

for r in run.results():
    print(r["rating"], r["questionText"], r["justification"])
    for scorer in r.get("codeScorerResults") or []:       # code scorers, trajectory/context match
        print(" ", scorer["name"], scorer["score"])

Each row carries rating/justification, the linked traceId, latency and token counts, similarity metrics, and every named scorer row. client.evaluations.get_run(run_id) fetches the same payload later, from a different process.

Pre-Deployment Quality Gate

Build & Run

Scoring & Datasets

Framework Examples

AI Analysis Report

Controlling the analysis

Fields

Reading it in your terminal

Per-result rows without an analysis

​Controlling the analysis

​Fields

​Reading it in your terminal

​Per-result rows without an analysis

Controlling the analysis

Fields

Reading it in your terminal

Per-result rows without an analysis