R099 · Agent principles and patterns › Evaluation data for agents

Evaluating agents at Amazon: three layers, traces and humans in the loop

Read the original

evalsagents-at-scalemulti-agentobservabilityhuman-oversightagent-reliability

TL;DR

Amazon's answer to "how do you evaluate thousands of agents" is to stop scoring only the final answer and score the trace: model, components (intent, planning, tools, memory) and outcome as three separate layers, so a failure can be localized to the layer that caused it. The source is an AWS Machine Learning blog post by Yunfei Bai, Allie Colin, Kashif Imran and Winnie Xiong (18 Feb 2026), resurfaced on LinkedIn by Sarthak Rastogi on 2026-11-15. It is a framework and metric catalog tied to Amazon Bedrock AgentCore Evaluations, not a study: no numbers on how well any metric predicts production quality.

The parts worth taking:

The framework

Two parts: a four-step evaluation workflow that is framework-agnostic (Strands, LangChain and LangGraph each ship their own eval modules, and teams did not want lock-in), and a metric library organized in three layers.

Workflow.

  1. Inputs are agent traces: offline traces uploaded after a run, or online traces with chosen dimensions and metrics.

  2. The library computes default and user-defined metrics over them.

  3. Results land in S3 or a dashboard alongside trace observability.

  4. Builders set degradation alerts, and schedule periodic human audits of a subset of traces and scores.

Three layers.

LayerQuestionExample metrics
Upper: outcomeDid the agent meet the goal, safely, at acceptable cost?Goal success, goal accuracy vs ground truth, correctness, helpfulness, relevance, conciseness, hallucination, toxicity, harmfulness
Middle: componentsWhich part of the agent broke?Intent detection; tool selection, parameter and error rate; multi-turn call sequence; memory context retrieval; topic adherence and refusal; grounding accuracy, reasoning faithfulness, per-step context score
Bottom: modelWhich model should power it?Benchmarks across foundation models for quality, latency and cost

The middle layer is the point. A final-answer score says the run failed; component scores say whether the plan, the tool pick, the arguments or the retrieval was wrong. The article also asks for evaluation of error recovery: invalid tool calls, malformed parameters, unexpected response formats, auth failures and memory misses, and whether the conversation stays coherent afterwards. It names these failure classes but gives no metric for recovery.

Many metric names match AgentCore Evaluations' built-ins; Amazon added use-case metrics on top. Several are LLM-judged (faithfulness, helpfulness, grounding), which is why judge calibration by humans shows up later as a requirement.

Three Amazon case studies

AgentWhat is evaluatedHow the eval data is madeLesson
Shopping assistant (hundreds to thousands of tools)Tool selection, parameter accuracy, multi-turn call sequenceGolden regression sets synthesized by an LLM from historical API invocation logsTool quality starts upstream: cross-org standards for tool signatures, input schemas, output contracts and descriptions; an LLM self-onboarding system generates them for legacy APIs
Customer service orchestratorIntent detection and routing to the right resolver subagent; task completion; topic adherenceAnonymized historical query and ground-truth intent pairs, replayed through an LLM simulator with virtual customer personasA wrong intent cascades: wrong resolver, irrelevant reply, escalation to human agents
Seller assistant (planner plus specialist agents)Planning score (subtask assignment), communication score, collaboration success rate, plus per-agent scoresNot statedCoordination failures, wrong decomposition and conflicting recommendations need human review; automated metrics miss emergent behavior

The four closing practices: evaluate quality, performance, responsibility and cost together (cost includes human effort and remediation); add use-case metrics such as first-contact resolution with domain experts; make HITL part of evaluation, not a fallback; and keep evaluating in production, because pre-deployment sets miss real usage and agents decay.

Post vs article

The LinkedIn post is a fair summary with two embellishments. "Replay traces against new versions" and "traces as first-class evaluation artifacts" are the post's framing: the article uses traces as evaluation input and builds regression sets from historical logs, but never describes replaying a recorded trajectory against a new agent version. "Generate synthetic eval datasets from production logs" is in the article, for the shopping assistant. The post also presents a February 2026 article as news.

Caveats

Review checklist

Sources

Connections

See R099 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.