Sources: Gunnar Morling's LinkedIn post (2026-11-11). Background: Hardwood 1.0 announcement and the beta release notes adding the S3 backend on morling.dev.
In one paragraph
Hardwood is an open-source, low-dependency Apache Parquet reader and writer for Java. When it began reading files directly from S3-compatible object storage, its author found the hard problem wasn't decoding (types, encodings, nesting, SIMD) but deciding what to do next. Each remote request can cost tens of milliseconds before the first byte arrives. A reader that fetches a chunk, decodes it and then fetches the next spends most of its time waiting. Hardwood now has a full work and IO scheduler. The stated goal: a read runs at the speed of its slowest resource (network, cores, memory) or of the code consuming rows, and never slower because of scheduling. The claim that generalizes is that any format read over a high-latency link, with dependencies between its parts, needs a scheduler.
Why Parquet over S3 is hard to schedule
The footer metadata sits at the end of the file, so the first useful fetch is the last bytes.
A page can be decoded only once three things are true: its bytes have arrived, its column chunk's dictionary is decoded, and its position in the output batch is known.
Some files have no page index, so pages are found only by scanning headers as bytes stream in.
Each of these steps completes in its own order on its own thread, so a simple fetch-then-decode loop leaves the network or the cores idle.
The design
Everything is a job. Fetching a byte range, parsing a footer, planning a row group and decoding a page are all jobs on one scheduler.
Priority by row. Each job is keyed by the first row it serves, and the lowest key runs first. The work the consumer will need soonest always wins.
Two executors. IO jobs run on Java 21 virtual threads, which are cheap enough to keep many requests in flight. CPU jobs (decoding) run on a worker pool of configurable size, or optionally on the caller's thread, for example in a single-core Kubernetes pod where extra threads would only compete.
Two read-ahead bounds. Fetch read-ahead and decode read-ahead are limited separately because they cost different things. Hiding latency needs many compressed bytes in flight, while decoded values are several times larger and need only a few batches of head start.
A progress guarantee. Whatever the consumer needs next always runs, regardless of the bounds, so the limits can never deadlock a read.
Zero-copy decode with buffer lifetimes. Pages decode straight from fetched buffers into output batches. The engine tracks the remaining readers of each buffer and frees it after the last page is done.
Fairness. Read-ahead is capped so memory stays bounded and one read can't starve another.
Tradeoffs and failure modes
Tuning surface. Two bounds and a pool size interact with object-store latency, row-group size and consumer speed. Defaults that suit a nearby region may under-fetch from a distant one.
Priority inversion and starvation. Lowest-row-first favors the head of the current read. Mixed workloads need the cross-read fairness the post mentions but doesn't detail.
Memory accounting. Reference-tracked buffers are only as safe as the tracking. A missed release leaks memory, and an early one corrupts data.
Request cost. More concurrent range requests lower latency but raise per-request charges and can hit provider throttling. The post doesn't discuss request coalescing.
Evidence. This is a design note from the author ahead of a promised longer write-up. No benchmarks are given for the scheduler itself.
Review checklist for a remote reader
Have you measured time to first byte per request from the regions you serve, and sized in-flight requests to hide it?
Do IO and CPU work run on separate executors with separate limits?
Is read-ahead bounded separately for compressed bytes and decoded data?
Is there an explicit rule that the consumer's next need always runs, so bounds can't deadlock the read?
Are buffer lifetimes tracked explicitly when decoding without copies?
Can the CPU work run on the caller's thread in constrained containers?
Is fairness across concurrent reads defined and tested?
Related
R001 (Scaling Git at OpenAI): another storage system redesigned around latency and access patterns, not raw parsing speed.
R035 (long-running agents and durable execution): scheduling dependent units of work so progress is guaranteed under bounded resources.