R046 · Infrastructure for agents at scale › Consistency and storage tradeoffs

Hardwood: a remote file reader is really a scheduler

Read the original

distributed-systemsdev-infra

Sources: Gunnar Morling's LinkedIn post (2026-11-11). Background: Hardwood 1.0 announcement and the beta release notes adding the S3 backend on morling.dev.

In one paragraph

Hardwood is an open-source, low-dependency Apache Parquet reader and writer for Java. When it began reading files directly from S3-compatible object storage, its author found the hard problem wasn't decoding (types, encodings, nesting, SIMD) but deciding what to do next. Each remote request can cost tens of milliseconds before the first byte arrives. A reader that fetches a chunk, decodes it and then fetches the next spends most of its time waiting. Hardwood now has a full work and IO scheduler. The stated goal: a read runs at the speed of its slowest resource (network, cores, memory) or of the code consuming rows, and never slower because of scheduling. The claim that generalizes is that any format read over a high-latency link, with dependencies between its parts, needs a scheduler.

Why Parquet over S3 is hard to schedule

The design

Tradeoffs and failure modes

Review checklist for a remote reader

Connections

See R046 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.