AI Analysis Report

.analyze() is the last step in the evaluation chain. It runs the same analysis as the dashboard’s “Analyze” button. On self-host the whole analysis runs synchronously in one request (the response reports mode: "sync"): the engine re-judges a sample of the rated results - the 12 worst plus the 5 best - with up to 3 judges, then produces the qualitative report in one final judge call, returned as a Report object.

report = client.evaluations.run(...).execute(my_agent).finalize().analyze(
    mode="auto",                                    # "auto" (default) | "sync" | "batch"
    quality_mode="balanced",                        # "balanced" (default) | "quality_first"
    judges=["gpt-5.6-luna", "claude-opus-5"],       # 1-3 model ids; omit for the platform default judge
)

report.summary                 # str | None, overall narrative summary
report.consistency_score       # float | None, 0-10, run-to-run consistency
report.instruction_adherence   # ReportInstructionAdherence | None
report.response_patterns       # ReportResponsePatterns | None
report.reasoning_analysis      # ReportReasoningAnalysis | None
report.tool_usage_analysis     # ReportToolUsageAnalysis | None
report.strengths               # list[str]
report.weaknesses              # list[str]
report.overall_rating          # str | None, "high" | "medium" | "low"
report.recommendations         # list[ReportRecommendation]

Numeric scores (average_rating, similarity metrics) are ready right after .finalize(), so you don’t need to wait for .analyze() just to see how the run went. Because .analyze() polls a job until it completes, it can take noticeably longer than a single LLM call for larger runs; progress is shown in the terminal while it waits.

Controlling the analysis

Parameter	Values	Description
`mode`	`"auto"` (default), `"sync"`, `"batch"`	How item scoring executes server-side. Self-host ignores this parameter entirely and always runs synchronously (the response reports `mode: "sync"`); on hosted AgentX, `"auto"` picks based on run size
`quality_mode`	`"balanced"` (default), `"quality_first"`	Recorded with the analysis request. On self-host it does not change scoring; on hosted AgentX, `"quality_first"` runs a second judge on every item while `"balanced"` samples based on risk
`judges`	1-3 model ids	Which LLM(s) score each response. On self-host, every requested judge scores every sampled item; hosted AgentX runs them in stages (the first always runs, a second confirms, a third breaks ties). Omit to score with a single judge, the engine’s platform default model
`poll_interval`	seconds, default `5.0`	How often to check job status while waiting
`timeout`	seconds, default `1800.0`	Give up waiting after this long (the job keeps running server-side; call `get_report()` later to check on it)

Check on a long-running analysis without calling .analyze() again, even from a separate script execution:

status = client.evaluations.get_analysis_status(run_id)
print(status.status, status.progress.overall_percentage)

Fields

Field	Shape	Description
`summary`	`str`	Overall narrative summary of the run
`consistency_score`	`float`	0-10, how consistent responses were across repeated runs of the same question
`instruction_adherence`	`{ score, analysis, deviations: [str], rating }`	How well responses followed `subject.agentInstructions`
`response_patterns`	`{ similarities: [str], differences: [str], outliers: [str], rating }`	Cross-run consistency patterns
`reasoning_analysis`	`{ cot_quality, reasoning_patterns: [str], reasoning_gaps: [str], rating }`	Quality of the agent’s chain-of-thought, when traced
`tool_usage_analysis`	`{ effectiveness, patterns: [str], issues: [str], rating }`	How well the agent used its tools, when tool calls were traced
`strengths`	`list[str]`	What the agent did well
`weaknesses`	`list[str]`	Where the agent fell short
`overall_rating`	`"high" \| "medium" \| "low"`	Coarse overall quality rating
`recommendations`	`[{ category, priority, recommendation, reasoning }]`	Actionable, prioritized fixes

Each rating sub-field is one of "high", "medium", "low". recommendations[].category is one of "instructions", "tools", "knowledge", "reasoning", "consistency", "other"; priority is "high", "medium", or "low".

Reading it in your terminal

report = client.evaluations.run(...).execute(my_agent).finalize().analyze()

print(f"Rating: {report.overall_rating}")
print(report.summary)

for rec in report.recommendations:
    print(f"[{rec.priority}] {rec.category}: {rec.recommendation}")

The full report, per-question breakdowns, and instruction-change suggestions are also in the dashboard: print run_context.run_id and open that run under Evaluate > Runs.

Per-result rows without an analysis

results() returns the scored rows - no analysis job, no extra judge calls - for asserting on individual results in scripts and CI. Rows are typed RunResultRow objects with snake_case attributes (dict-style access still works but is deprecated and warns; row.raw keeps the full wire dict):

run = client.evaluations.run(...).execute(my_agent).finalize()

for r in run.results():
    print(r.rating, r.question_text, r.justification)
    for scorer in r.code_scorer_results or []:            # code scorers, trajectory/context match
        print(" ", scorer["name"], scorer["score"])

Each row carries rating/justification, a status ("scored", "skipped" when the judge could not score it - rating stays None, not 0 - or "failed" when the result carried an error). A failed row is stored with rating 0 and IS counted into averageRating/ratedCount and any CI gate, while a skipped row stays None and is not. Rows also carry the linked trace_id, latency_ms and token counts, and the similarity metrics (cosine_similarity, jaccard_similarity, bleu_score, rouge_score). client.evaluations.get_run(run_id) fetches the same payload later, from a different process.

Pre-Deployment Quality Gate

Build & Run

Datasets & RAG

Playground

Framework Examples

AI Analysis Report

Controlling the analysis

Fields

Reading it in your terminal

Per-result rows without an analysis

​Controlling the analysis

​Fields

​Reading it in your terminal

​Per-result rows without an analysis

Controlling the analysis

Fields

Reading it in your terminal

Per-result rows without an analysis