Much of LLM serving performance is won or lost around the model, not inside it. Pallavi Ahuja's post (2026-11-12) lists six techniques that each remove a different kind of waste while the weights and the math stay exactly the same.
The claim
The post's point is that knowing how attention works is not the same as knowing why a deployment is slow or expensive. The six techniques are scheduling, memory management and placement decisions. None of them changes model quality: prefix caching, continuous batching, PagedAttention, chunked prefill and disaggregation are exact, and speculative decoding with the standard accept/reject rule keeps the target model's output distribution.
The claim holds for latency, throughput and cost per token. It is a primer: no numbers, benchmarks or citations, and each idea maps to published work (listed under Sources). The value for a staff engineer is the framing: each technique targets a named waste, so you can tell which one your workload actually has before turning it on.
The six techniques
Each row is one kind of waste. Prefill (reading the prompt) is mostly compute-bound; decode (one token per step) is mostly memory-bandwidth-bound, and most of the table follows from that split.
| Technique | Waste removed | Mechanism | Cost or tradeoff | Pays off when |
|---|---|---|---|---|
| Prefix caching | Recomputing the same prompt prefix in prefill | Keep KV states for a shared token prefix and reuse them; the rest of the prompt and all decoding run as usual | Cache memory competes with live batches; a hit needs a byte-identical prefix | Prompts actually repeat: system prompts, tool schemas, RAG templates, multi-turn chat |
| Continuous batching | GPU idle while a fixed batch waits for its longest request | Between iterations, retire finished sequences and admit waiting ones | More scheduler work; per-step latency varies with batch makeup | Mixed output lengths, which is almost every real workload |
| PagedAttention | KV memory lost to fragmentation and worst-case reservation | Store KV in fixed blocks; a block table maps logical blocks to non-contiguous physical ones | Kernel indirection; block size is a tuning knob. Manages memory only, never approximates attention | Variable sequence lengths; it is what lets the other techniques fit more requests |
| Chunked prefill | Decode steps stalled behind one long prompt | Split a long prefill into chunks and interleave them with decode steps | Chunk size trades time-to-first-token against inter-token latency and throughput | Long prompts share GPUs with interactive traffic |
| Speculative decoding | One full target-model pass per output token | A cheap draft proposes several tokens; the target verifies them in one pass and keeps the accepted run | Draft cost, verification cost and acceptance rate set the gain; it can lose at large batch sizes where decode is no longer bandwidth-bound | Low-batch, latency-sensitive decode with predictable text (code, structured output) |
| Prefill/decode disaggregation | Compute-bound and bandwidth-bound phases fighting over the same GPUs | Separate prefill and decode worker pools, each sized and parallelized for its phase; KV moves between them | KV transfer cost (needs fast interconnect), two pools to capacity-plan, more failure points | Large fleets with strict time-to-first-token and inter-token SLOs at once |
How they interact and fail
The six are not independent switches. They share one budget (GPU memory for KV) and one scheduler, so turning one on changes the others' payoff.
PagedAttention is the enabler. Block-level KV management is what makes prefix sharing cheap (shared blocks, copy-on-write) and lets continuous batching admit more sequences. Without it, the others fight over contiguous reservations.
Head-of-line blocking is the common production failure. A commenter on the post saw latency spikes from one long prefill queuing every request behind it. Continuous batching alone does not fix this; chunked prefill (or disaggregation) does.
Prefix caching only pays when prompts repeat. The same commenter's point: hit rate is a workload property. Prompt layout decides it (stable content first, volatile last), so it is partly an application concern, not only a server flag.
Speculative decoding can make things worse. At high batch sizes decode becomes compute-bound and draft verification is extra work. Low acceptance rates (creative text, a poorly matched draft) also erase the gain. Measure accepted tokens per step before trusting it.
Disaggregation moves the bottleneck to the network. KV for a long prompt is large; transfer time and a second capacity plan are the price. It is a fleet-scale decision, not a default.
Preemption and eviction are where correctness meets performance. When KV memory runs out, sequences are swapped or recomputed, and cached prefixes are evicted. These paths are rarely exercised in tests and show up as tail-latency cliffs.
Another commenter's rule of thumb, that continuous batching, PagedAttention and prefix caching cover about 90% of workloads, matches what mainstream engines (vLLM, SGLang, TensorRT-LLM) enable by default. Treat chunked prefill, speculation and disaggregation as targeted fixes for a measured problem.
Review checklist for a serving stack
Do you measure time-to-first-token and inter-token latency separately, at p50 and p99? Most of these techniques move one at the expense of the other.
What is the prefix-cache hit rate per route, and is the prompt ordered stable-to-volatile so hits are possible?
Is continuous batching on, and what caps admission (max sequences, max batched tokens)?
How much GPU memory goes to KV, and what happens on exhaustion: preempt, swap or recompute? Is that path load-tested?
Is there a prompt-length distribution? If long prompts share GPUs with chat traffic, is chunked prefill on and the chunk size tuned?
If speculative decoding is on, what are the acceptance rate and tokens accepted per step at your real batch size?
Before disaggregating: is the interconnect fast enough for KV transfer, and do you have SLOs on both phases that a single pool cannot meet?
On a hosted provider: which of these does it run, and does its quantization or tier change output quality (see R027)?
Related entries
R027 (the Tool-call index): the same model behaves differently by provider; serving choices like quantization and tier are part of why.
R026 (HybridAgent): treats the KV cache as cross-turn working memory, which makes cache eviction and transfer a state-management problem, not only a performance one.
R046 (Hardwood): the same lesson in a different system. A remote reader is really a scheduler, and so is an inference server.
R032 (Strands Decider): the other way to cut serving cost, by replacing a generative call with a one-pass decision model.
Sources
Pallavi Ahuja, LinkedIn post, 2026-11-12, with its comments.
Background papers: Kwon et al., PagedAttention / vLLM, SOSP 2023; Yu et al., Orca, OSDI 2022; Agrawal et al., Sarathi-Serve, OSDI 2024; Leviathan et al., speculative decoding, ICML 2023; Chen et al., speculative sampling, 2023; Zhong et al., DistServe, OSDI 2024; Patel et al., Splitwise, ISCA 2024.