R080 · AI-built software and its verification › Test oracles for agent-written code

AWS Emulator: a replica graded by the real service

Read the original

test-oraclesai-built-softwareservice-emulationsandboxingdev-infraagents-at-scale

Source: André Lindenberg's LinkedIn post (2026-11-13) on aws/aws-emulator.

TL;DR

AWS has released AWS Emulator, a local program that serves the AWS APIs so code and coding agents can test against it with no AWS account and no calls to AWS. The post's point is the build method, not the product: more than 30,000 commits were made by agents, and the test suite was first corrected until real AWS passed it, then the emulator was built until it passed the same tests. Real AWS is the test oracle. As Lindenberg puts it, "A replica earns trust when the real service grades its tests." Fidelity gaps are published, per service and per behavior.

What it is

The method: real AWS as the oracle

The usual way to build a fake is to write the fake and tests together, which lets the two agree with each other and both be wrong. The described process breaks that loop in two phases:

  1. Calibrate the tests. Agents run the test suite against real AWS and fix the tests (not the service) until real AWS passes. Now each test encodes observed AWS behavior, not someone's reading of the docs.

  2. Build to the tests. Agents then implement the emulator until it passes those calibrated tests. The tests are frozen in this phase; the emulator is the thing that changes.

The Smithy models pin the wire format mechanically, so agent effort goes into semantics: state, errors, pagination, eventual-consistency quirks.

Fidelity is treated as data. Each release publishes a fidelity score per service, and known gaps are files in the repo (issues/things-that-do-not-work/), moving to issues/things-that-now-work/ when fixed. A user can check whether their project should work before adopting it.

Why it matters for staff engineers

Tradeoffs and failure modes

Review checklist

Adopting the emulator:

Applying the method to your own agent-built replica or migration:

Connections

See R080 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.