R051 · Infrastructure for agents at scale › Inference providers and serving reliability

Six LLM serving techniques that leave the model alone

Read the original

llm-servinginference-costprompt-cachingdev-infra

Much of LLM serving performance is won or lost around the model, not inside it. Pallavi Ahuja's post (2026-11-12) lists six techniques that each remove a different kind of waste while the weights and the math stay exactly the same.

The claim

The post's point is that knowing how attention works is not the same as knowing why a deployment is slow or expensive. The six techniques are scheduling, memory management and placement decisions. None of them changes model quality: prefix caching, continuous batching, PagedAttention, chunked prefill and disaggregation are exact, and speculative decoding with the standard accept/reject rule keeps the target model's output distribution.

The claim holds for latency, throughput and cost per token. It is a primer: no numbers, benchmarks or citations, and each idea maps to published work (listed under Sources). The value for a staff engineer is the framing: each technique targets a named waste, so you can tell which one your workload actually has before turning it on.

The six techniques

Each row is one kind of waste. Prefill (reading the prompt) is mostly compute-bound; decode (one token per step) is mostly memory-bandwidth-bound, and most of the table follows from that split.

TechniqueWaste removedMechanismCost or tradeoffPays off when
Prefix cachingRecomputing the same prompt prefix in prefillKeep KV states for a shared token prefix and reuse them; the rest of the prompt and all decoding run as usualCache memory competes with live batches; a hit needs a byte-identical prefixPrompts actually repeat: system prompts, tool schemas, RAG templates, multi-turn chat
Continuous batchingGPU idle while a fixed batch waits for its longest requestBetween iterations, retire finished sequences and admit waiting onesMore scheduler work; per-step latency varies with batch makeupMixed output lengths, which is almost every real workload
PagedAttentionKV memory lost to fragmentation and worst-case reservationStore KV in fixed blocks; a block table maps logical blocks to non-contiguous physical onesKernel indirection; block size is a tuning knob. Manages memory only, never approximates attentionVariable sequence lengths; it is what lets the other techniques fit more requests
Chunked prefillDecode steps stalled behind one long promptSplit a long prefill into chunks and interleave them with decode stepsChunk size trades time-to-first-token against inter-token latency and throughputLong prompts share GPUs with interactive traffic
Speculative decodingOne full target-model pass per output tokenA cheap draft proposes several tokens; the target verifies them in one pass and keeps the accepted runDraft cost, verification cost and acceptance rate set the gain; it can lose at large batch sizes where decode is no longer bandwidth-boundLow-batch, latency-sensitive decode with predictable text (code, structured output)
Prefill/decode disaggregationCompute-bound and bandwidth-bound phases fighting over the same GPUsSeparate prefill and decode worker pools, each sized and parallelized for its phase; KV moves between themKV transfer cost (needs fast interconnect), two pools to capacity-plan, more failure pointsLarge fleets with strict time-to-first-token and inter-token SLOs at once

How they interact and fail

The six are not independent switches. They share one budget (GPU memory for KV) and one scheduler, so turning one on changes the others' payoff.

Another commenter's rule of thumb, that continuous batching, PagedAttention and prefix caching cover about 90% of workloads, matches what mainstream engines (vLLM, SGLang, TensorRT-LLM) enable by default. Treat chunked prefill, speculation and disaggregation as targeted fixes for a measured problem.

Review checklist for a serving stack

Sources

Connections

See R051 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.