TL;DR
Amazon's answer to "how do you evaluate thousands of agents" is to stop scoring only the final answer and score the trace: model, components (intent, planning, tools, memory) and outcome as three separate layers, so a failure can be localized to the layer that caused it. The source is an AWS Machine Learning blog post by Yunfei Bai, Allie Colin, Kashif Imran and Winnie Xiong (18 Feb 2026), resurfaced on LinkedIn by Sarthak Rastogi on 2026-11-15. It is a framework and metric catalog tied to Amazon Bedrock AgentCore Evaluations, not a study: no numbers on how well any metric predicts production quality.
The parts worth taking:
Split tool metrics. Tool selection accuracy (right tool?), tool parameter accuracy (right arguments?), tool call error rate (did it run?) and multi-turn call-sequence accuracy need different fixes, so they are reported separately.
Tool schemas are governed, not ad hoc. Bad descriptions cause wrong tool picks, which bloat context and cost. Amazon set cross-org standards for tool schemas and descriptions and uses an LLM to generate them when onboarding APIs.
Golden sets come from production. Regression sets are synthesized by an LLM from historical API invocation logs; intent sets pair anonymized real queries with ground-truth intents and run through LLM-simulated customer personas.
Multi-agent gets its own scores. Planning score (subtask assignment), communication score, collaboration success rate, plus human review of coordination failures.
Humans are in the loop by design: they label golden sets, calibrate LLM-as-judge and audit sampled traces on a schedule.
The framework
Two parts: a four-step evaluation workflow that is framework-agnostic (Strands, LangChain and LangGraph each ship their own eval modules, and teams did not want lock-in), and a metric library organized in three layers.
Workflow.
Inputs are agent traces: offline traces uploaded after a run, or online traces with chosen dimensions and metrics.
The library computes default and user-defined metrics over them.
Results land in S3 or a dashboard alongside trace observability.
Builders set degradation alerts, and schedule periodic human audits of a subset of traces and scores.
Three layers.
| Layer | Question | Example metrics |
|---|---|---|
| Upper: outcome | Did the agent meet the goal, safely, at acceptable cost? | Goal success, goal accuracy vs ground truth, correctness, helpfulness, relevance, conciseness, hallucination, toxicity, harmfulness |
| Middle: components | Which part of the agent broke? | Intent detection; tool selection, parameter and error rate; multi-turn call sequence; memory context retrieval; topic adherence and refusal; grounding accuracy, reasoning faithfulness, per-step context score |
| Bottom: model | Which model should power it? | Benchmarks across foundation models for quality, latency and cost |
The middle layer is the point. A final-answer score says the run failed; component scores say whether the plan, the tool pick, the arguments or the retrieval was wrong. The article also asks for evaluation of error recovery: invalid tool calls, malformed parameters, unexpected response formats, auth failures and memory misses, and whether the conversation stays coherent afterwards. It names these failure classes but gives no metric for recovery.
Many metric names match AgentCore Evaluations' built-ins; Amazon added use-case metrics on top. Several are LLM-judged (faithfulness, helpfulness, grounding), which is why judge calibration by humans shows up later as a requirement.
Three Amazon case studies
| Agent | What is evaluated | How the eval data is made | Lesson |
|---|---|---|---|
| Shopping assistant (hundreds to thousands of tools) | Tool selection, parameter accuracy, multi-turn call sequence | Golden regression sets synthesized by an LLM from historical API invocation logs | Tool quality starts upstream: cross-org standards for tool signatures, input schemas, output contracts and descriptions; an LLM self-onboarding system generates them for legacy APIs |
| Customer service orchestrator | Intent detection and routing to the right resolver subagent; task completion; topic adherence | Anonymized historical query and ground-truth intent pairs, replayed through an LLM simulator with virtual customer personas | A wrong intent cascades: wrong resolver, irrelevant reply, escalation to human agents |
| Seller assistant (planner plus specialist agents) | Planning score (subtask assignment), communication score, collaboration success rate, plus per-agent scores | Not stated | Coordination failures, wrong decomposition and conflicting recommendations need human review; automated metrics miss emergent behavior |
The four closing practices: evaluate quality, performance, responsibility and cost together (cost includes human effort and remediation); add use-case metrics such as first-contact resolution with domain experts; make HITL part of evaluation, not a fallback; and keep evaluating in production, because pre-deployment sets miss real usage and agents decay.
Post vs article
The LinkedIn post is a fair summary with two embellishments. "Replay traces against new versions" and "traces as first-class evaluation artifacts" are the post's framing: the article uses traces as evaluation input and builds regression sets from historical logs, but never describes replaying a recorded trajectory against a new agent version. "Generate synthetic eval datasets from production logs" is in the article, for the shopping assistant. The post also presents a February 2026 article as news.
Caveats
No evidence of validity. No numbers on how well any metric, judge or synthetic set predicts real customer outcomes, and no before/after results from the case studies.
Vendor framing. The metric list is largely AgentCore Evaluations' catalog; the framework-agnostic claim sits next to a single-vendor implementation.
LLM grading LLMs. Golden sets are LLM-synthesized from logs, personas are LLM-simulated, and many scores are LLM-judged. Human calibration is required, but the article gives no agreement rates or audit sample sizes.
Replay goes stale. A commenter asks how long a recorded trace stays valid once tools change. The article does not answer; schema standards help, but a changed tool contract invalidates the golden set.
Recovery is named, not measured. Error-recovery scenarios are listed without a metric.
Review checklist
Do we score tool selection, tool arguments and tool execution separately?
Is there a written standard for tool names, schemas, output contracts and descriptions, and is it enforced at onboarding?
Are regression sets drawn from real traffic, refreshed when tool contracts change, and versioned with them?
For orchestrators: is routing or intent accuracy measured against ground truth, apart from final answer quality?
For multi-agent: are decomposition, assignment and handoffs scored, and are coordination failures sampled for human review?
Is every LLM judge calibrated against human labels, with agreement tracked over time?
Are production scores alerting on degradation, with a scheduled human audit of sampled traces?
Is cost (tokens, tool calls, human remediation) reported next to quality?
Related
R069: a judge model scoring every step of a trajectory, the middle layer done by a critic.
R064: SkillEvaluator's paired runs measure whether a component helps, rather than scoring it in isolation.
R029 and R070: synthetic environments as eval data when production logs can't be used.
R039 and R066: noise floors and statistical acceptance before trusting a score change.
R016 and R020: human checking and harness-enforced gates; why humans stay in the loop.
R027: tool-call failure rates differ by provider for the same model, a bottom-layer effect that looks like a middle-layer failure.
R005 and R006: the layered mental model and the harness vocabulary this framework evaluates.
Sources
Evaluating AI agents: Real-world lessons from building agentic systems at Amazon, Yunfei Bai, Allie Colin, Kashif Imran and Winnie Xiong, AWS Machine Learning Blog, 18 Feb 2026.
Sarthak Rastogi's LinkedIn post, 2026-11-15.