Coming from DeepEval

DeepEval is a metrics library; AgentX is a self-host evaluation platform. If you’ve built evals on DeepEval, almost everything you wrote maps directly - and the platform half (live scoring, review, calibration, dashboards) is what you gain. This page is the honest mapping, based on a side-by-side run of identical test cases through both systems.

Metric mapping

DeepEval	AgentX	Notes
`FaithfulnessMetric`	RAG: Faithfulness	AgentX is stricter by design: claims must be supported by the context, not merely non-contradicting. In our comparison, a fully unsupported answer scored 1.00 on DeepEval (“no contradictions”) and 1/10 here.
`AnswerRelevancyMetric`	RAG: Answer Relevancy	Both judge direction only, never correctness (that’s Faithfulness’s job).
`ContextualRelevancyMetric`	RAG: Context Relevancy	AgentX judges holistically with per-chunk reasoning; DeepEval scores a statement ratio, which can cliff (a retrieval containing the exact answer scored 0.00 in our comparison when a sibling chunk diluted the ratio).
`ContextualPrecisionMetric`	RAG: Contextual Precision	Same contract, including “no relevant chunks scores 0”.
`ContextualRecallMetric`	RAG: Contextual Recall	Both require a reference answer. DeepEval raises `MissingTestCaseParamsError` mid-run without one; AgentX marks the scorer Needs a reference answer - live scoring is refused up front and reference-less dataset cases skip with a reason.
`GEval(criteria=...)`	Any judge scorer	Your GEval criteria paste straight into a scorer’s acceptance/rejection/evaluation criteria - and the same scorer then also scores live traffic.
`TaskCompletionMetric`	Agent: Task Completion	Trajectory-anchored: the judge sees the tools called, in order, with failures.
`ToolCorrectnessMetric`	Agent: Tool Correctness + `expectedTools` on cases	For exact matching, dataset cases’ `expected_tools` is deterministic - zero LLM cost. The judge template covers the semantic cases exact matching can’t.
`StepEfficiencyMetric`	Agent: Step Efficiency	Loops, redundant calls, dead ends - judged from the trajectory.
`KnowledgeRetentionMetric`	Session: Knowledge Retention	Enable live scoring with per-session scope; the judge grades the whole transcript.
`RoleAdherenceMetric`	Session: Role Adherence	Same session mechanism.
`ToxicityMetric` / `BiasMetric`	Safety: Harmful Content / Safety: Bias & Fairness	Judge-based, meaning-level. The zero-cost template patterns (secrets, PII, profanity) catch literal matches without any LLM spend.
`assert_test` (pytest)	`agentx.testing.assert_evaluation`	Same ergonomic; the verdict is also recorded in the dashboard’s CI gate history.

What works differently (on purpose)

One scorer, both surfaces. A DeepEval metric is an offline construct; online evaluation lives in its Confident AI cloud. An AgentX judge scorer grades dataset runs and live traffic with one rubric - see the two grading modes.
0-10 with prose, not 0-1 ratios. DeepEval scores are claim/statement ratios (auditable arithmetic, occasional cliffs). AgentX judges holistically and states its counts in the justification (“2 of 3 claims supported”) - auditable prose, no cliffs.
Hallucinations fail hard. DeepEval’s ratio math gave a known contradiction 0.50 in our comparison; AgentX’s exact-fact rule floors a wrong core fact to 0-3 regardless of polish.

What we don’t have (and where to go instead)

Academic benchmarks (MMLU, HumanEval, GSM8K…): DeepEval ships 17. These benchmark foundation models, not your agents - if you need them, run DeepEval’s benchmark suite; it coexists fine with AgentX in the same codebase.
Dataset synthesizer: AgentX builds datasets from production traces and generates grounded cases from pasted source text (plain text or Markdown only); DeepEval’s synthesizer adds document-file parsing and larger-scale generation.
Red-teaming: use their deepteam; AgentX’s safety scorers monitor production, they don’t attack it.
Multimodal metrics (image, audio, voice): DeepEval only.

Migration in practice

# DeepEval
metric = GEval(name="Helpfulness", criteria="Answers using the retrieved policy...")
assert_test(LLMTestCase(input=q, actual_output=a), [metric])

# AgentX - same rubric, plus it now also scores live traffic
scorer = client.monitor.judge_scorers.create(
    name="Helpfulness",
    judge={"acceptanceCriteria": "Answers using the retrieved policy..."},
)
report = client.evaluations.run(
    dataset_id=ds,
    subject={"kind": "custom_agent", "displayName": "Helpfulness Agent", "framework": "other"},
    scorer_id=scorer.id,
).execute(agent).finalize()
assert_evaluation(report, min_rating=7.0, no_regression=True)

Get Started

Self-Hosting

Python SDK

Guides

Coming from DeepEval

Metric mapping

What works differently (on purpose)

What we don’t have (and where to go instead)

Migration in practice

​Metric mapping

​What works differently (on purpose)

​What we don’t have (and where to go instead)

​Migration in practice

Metric mapping

What works differently (on purpose)

What we don’t have (and where to go instead)

Migration in practice