R102 · Agent principles and patterns › Evaluation data for agents

AI-assisted performance tuning overfits to the benchmark machine

Read the original

coding-agentsperformancebenchmarkstestingevaluation

Source: Gunnar Morling (creator of Hardwood, the One Billion Row Challenge and MapStruct; former Debezium lead), LinkedIn, 2026-11-16, drawing on work on Hardwood, a dependency-light Parquet reader and writer for the JVM (hardwood-hq/hardwood).

The claim

Coding agents are very good at closing a measured performance gap, and that is the problem. Given one benchmark on one machine, an agent will happily add fast paths for the data shapes in the test set and size data structures to that CPU's cache lines. For an application, where you deploy to the hardware you benchmark on, that is legitimate tuning. For a library that runs on hardware and data you have never seen, it is overfitting. Morling reports seeing agent-proposed changes in Hardwood that help on x86 and hurt on aarch64, or the reverse. His remedy: a broad benchmark matrix matters more than ever.

The Hardwood repository itself shows how easy the trap is: its published headline read benchmark is a single run configuration (an Apple M3 Max, aarch64, Java 25, one public taxi dataset). That is a fine progress signal, and exactly the kind of single target an optimizing agent will chase.

Why agents make this worse than humans

What a broad benchmark matrix should cover

Tradeoffs

Failure modes to watch

Review checklist for agent-authored performance changes

Connections

See R102 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.