07. Prefill vs Decode — Why One GPU Can't Do Both Well

The Problem

When you send a prompt to an LLM, two very different things happen. To understand why, you need to know one thing about how transformer-based LLMs generate text: they predict one token at a time, and each token depends on everything before it.

Quick primer — Prefill and Decode are transformer-specific.

These two phases exist because of how attention works in transformers. The model needs to "look at" all previous tokens to predict the next one. For each token, it stores a Key ("here's what I contain") and a Value ("here's what I contribute"). To generate a new token, the model computes a Query ("what am I looking for?") and matches it against ALL stored Keys to decide which Values to use.

Think of it like a search engine: the Query is your search, Keys are page titles (used for ranking), Values are page content (what you read after clicking). The KV cache is like a search index that grows with every word generated — and you have to scan the entire index for every single new word.

Here's a concrete example. You type: "Explain quantum computing in simple terms"

Phase 1: Prefill (process the input)

Input:  [Explain] [quantum] [computing] [in] [simple] [terms]
                         ↓
        Process ALL 6 tokens in one parallel operation
                         ↓
Output: KV cache for all 6 tokens + prediction for token 7

All 6 tokens exist already, so their Keys and Values can be computed simultaneously — one big matrix multiply. This is compute-bound. The GPU's compute units are fully saturated doing math. It's fast (milliseconds to seconds), predictable, and looks a lot like a traditional ML batch inference call.

Phase 2: Decode (generate the response)

Step 1: Read ALL Keys/Values → compute attention → generate "Quantum"
        Store "Quantum"'s Key and Value into cache

Step 2: Read ALL Keys/Values (now 7) → generate "computing"
        Store "computing"'s Key and Value into cache

Step 3: Read ALL Keys/Values (now 8) → generate "is"
        ...repeat 500 more times...

Each step produces ONE token. The math per step is tiny (one token's worth of computation). But the model must read the entire KV cache (all prior tokens' Keys/Values) plus all model weights from GPU memory (HBM) — just to produce that one token. The GPU spends most of its time waiting for memory reads to complete, not doing math.

For a 70-billion-parameter model generating the 500th token: the actual arithmetic takes ~0.14ms. Reading 140GB of weights from memory takes ~42ms. The GPU is idle 99.7% of the time, waiting for data. That's what "memory-bandwidth-bound" means.

Here's the mismatch in numbers (rough, for a 70B-parameter model on an H100 GPU):

These two phases have completely different resource profiles. Putting them on the same GPU is like running an OLTP workload (lots of small random reads, latency-sensitive) and an OLAP workload (big sequential scans, throughput-sensitive) on the same database instance. It "works" but you're always compromising one for the other.

Why Co-location Hurts

When prefill and decode share a GPU, bad things happen:

Decode gets starved. A decode request is generating tokens every 30-50ms. Then a new prefill request arrives — a big parallel matrix multiply that hogs the compute units for 200ms. During that 200ms, all active decode requests are stalled. Their TPOT spikes. Users see the response "freeze" mid-generation.

If you've ever seen a database query spike because a big analytical scan competed with your transaction workload — same dynamic. One heavy operation starves the latency-sensitive ones.

You can't optimize parallelism for both. Prefill wants high tensor parallelism (split one layer's computation across many GPUs for faster math). Decode wants less tensor parallelism (more GPUs means more synchronization overhead per token, which hurts latency on an already memory-bound workload). On a shared GPU pool, you pick one strategy — and it's wrong for the other phase.

SLOs conflict. You have two SLOs: TTFT (how fast the first token appears — prefill latency) and TPOT (how fast subsequent tokens stream — decode latency). Optimizing for one hurts the other. Pack more prefill work to improve TTFT throughput? You inflate TPOT. Protect decode latency? Prefill queues up and TTFT degrades.

DistServe (OSDI 2024) named this cleanly: they defined "goodput" as the maximum request rate where BOTH SLOs are met simultaneously. Under co-location, goodput was dramatically lower than raw throughput — because one SLO was always being violated.

The Solution: Disaggregation

The fix is conceptually simple: run prefill and decode on separate GPU pools.

Prefill nodes get optimized for compute throughput — high parallelism (split layers across many GPUs), big batches of prompts processed in parallel, maximum compute utilization.

Decode nodes get optimized for memory bandwidth — lower parallelism, maximum concurrent sequences, KV cache packed tight.

After prefill completes, the KV cache gets transferred to a decode node, and generation begins there.

Three teams arrived at this independently in 2024, each with a different emphasis:

DistServe (OSDI 2024) — Optimize parallelism per phase

Gave each phase its own independently-tuned parallelism strategy. Prefill workers use tensor parallelism (split each layer across GPUs for faster compute). Decode workers use pipeline parallelism (each GPU handles different layers sequentially, reducing per-token synchronization overhead). A placement algorithm co-locates prefill/decode pairs on nodes sharing NVLink (a high-bandwidth GPU-to-GPU interconnect, ~900 GB/s) so KV transfer is fast.

KV transfer cost: ~17.6ms for 2048 tokens on a 175B-parameter model over PCIe 5.0. That's less than a single decode step. Over NVLink, it's negligible.

Result: 7.4x more requests meeting SLOs, or 12.6x tighter SLO achievable vs. co-located vLLM (the standard open-source LLM serving engine).

Splitwise (ISCA 2024, Microsoft) — Optimize hardware cost

Went further: if decode is memory-bound and barely uses compute, why put it on expensive H100s? Run prefill on top-tier GPUs (maximum compute power). Run decode on cheaper, older, or lower-power hardware that still has good memory bandwidth.

Result: 1.4x throughput at 20% lower cost. Or 2.35x throughput within the same budget.

This is the same logic behind using SSDs for hot data and spinning disk for cold data — match the hardware to the workload profile, don't over-spec the cheap tier.

Mooncake (Moonshot AI, 2024) — Optimize around the cache

Mooncake flipped the framing entirely. Instead of asking "where should compute run?", they asked "where does the KV cache already live?"

The KV cache becomes a first-class distributed resource — a shared pool spanning GPU HBM, CPU DRAM, and SSDs across the cluster. The scheduler routes requests based on cache locality: if a user's conversation history is already cached on node X, route the next turn there. Don't transfer — just reuse.

This is the CDN insight: route to where the data already is, not where the compute is idle.

Result: 75% more requests in production (Kimi's actual serving platform), 525% in high-reuse simulation scenarios.

The Twist: First-Gen Disaggregation Assumed Simple Workloads

All three 2024 papers assumed a clean pattern: one prefill → one decode → done.

Real workloads in 2025-2026 don't look like that:

• Multi-turn conversations: User sends message, model responds, user sends another message. The second turn needs a small "incremental prefill" (just the new user message), not a full prefill from scratch.

• Agentic loops: Model calls a tool, gets a result, needs to process it and continue generating. Repeated small prefill-decode-prefill-decode cycles.

• RAG pipelines: Retrieve documents mid-generation, inject them as additional context, continue.

In these workloads, you're constantly switching between tiny prefills and decode steps. Shipping KV cache back and forth between prefill and decode pools for every micro-turn is insane — the transfer overhead alone would kill your latency.

This is where 2025-2026 research picks up.

The Evolution: Adaptive Disaggregation (2025-2026)

AMPD (2026) — "Where should incremental prefill run?"

For multi-round workloads, AMPD makes a per-request runtime decision: should this incremental prefill (processing new user input in turn 2, 3, 4...) go to the prefill pool, or just run locally on the decode node that already holds the KV cache?

The tradeoff:

• Send to prefill pool: Faster compute, but requires KV cache sync and adds network latency

• Run on decode node: Slower compute (decode nodes aren't optimized for prefill), but zero transfer overhead and the KV cache is already there

AMPD decides dynamically based on real-time load — if prefill nodes are idle, use them. If the network is congested, run locally. The planning algorithm jointly optimizes resource allocation across both phases.

This is capacity-aware routing with real-time feedback — same pattern as smart load balancers that route based on backend health rather than round-robin.

"Not All Prefills Are Equal" / PPD (2026) — Three roles, not two

Goes further by introducing a third node type: "Prefill-capable Decode" nodes that can handle small append-prefills without the interference problems of full prefills.

The insight: appending 200 new tokens to an existing 4000-token KV cache is a fundamentally different operation than prefilling 4200 tokens from scratch. The append is small enough that it doesn't starve concurrent decode work the way a full prefill would.

So for multi-turn:

• Turn 1 (full prefill): always goes to dedicated prefill nodes

• Turn 2+ (append-prefill): dynamically routed — either to prefill nodes OR handled locally on decode nodes

Result: ~68% reduction in Turn 2+ TTFT with competitive TPOT maintained. The network congestion problem from constant KV transfers effectively disappears for multi-turn workloads.

TaiChi (2025) — "Sometimes co-location is actually better"

Controversial take: pure disaggregation isn't always optimal.

When your TTFT constraint is very tight (users notice if the first token takes more than 200ms), disaggregation adds queuing delay — the request has to wait for a prefill node, then wait for KV transfer, then wait for decode node. Co-location eliminates all that queuing.

TaiChi's solution: don't choose between disaggregation and co-location. Deploy two instance types simultaneously:

• Prefill-heavy instances: Fast prefill, but decode suffers some interference

• Decode-heavy instances: Clean decode, but slower prefill

Three configurable knobs control the ratio and routing between them. The system smoothly interpolates between pure aggregation and pure disaggregation based on which SLO is tighter.

Result: 77% goodput improvement over SOTA under balanced SLOs. Matches pure disaggregation when TPOT is tight, matches pure co-location when TTFT is tight.

This is the "hybrid OLTP/OLAP" approach — some databases run both workloads on the same engine with careful resource isolation, rather than maintaining entirely separate systems. Sometimes the overhead of separation isn't worth it.

EcoInfer (2026) — "Decode wastes power"

A different angle entirely: if decode barely uses the GPU's compute units, why are we running the GPU at full clock speed during decode?

EcoInfer applies Dynamic Voltage and Frequency Scaling (DVFS) at iteration granularity:

• During prefill iterations: full GPU clock speed (compute-bound, need maximum math throughput)

• During decode iterations: lower clock speed (memory-bound, compute is idle anyway)

Lowering frequency during decode doesn't measurably hurt latency (you're waiting on memory, not compute) but saves significant power.

The related throttLL'eM paper reports up to 43.8% energy reduction with a similar approach. At datacenter scale with thousands of GPUs running decode 80% of the time, that's a meaningful electricity bill reduction.

If you've worked with CPU frequency governors (ondemand, powersave, performance) — this is the GPU equivalent, but at per-iteration granularity rather than per-second. The workload characteristics change fast enough that you need to switch modes hundreds of times per second.

The Big Picture

The evolution of prefill/decode disaggregation in 3 years:

2024: Binary split (prefill pool / decode pool)
      └─ Clean, simple, 7x better goodput

2025: Hybrid (sometimes co-locate when it's better)
      └─ TaiChi: 77% improvement by not dogmatically disaggregating

2026: Adaptive (decide per-request, per-turn, per-iteration)
      └─ AMPD, PPD: handle real agentic workloads
      └─ EcoInfer: optimize power at iteration granularity

The pattern is familiar from any distributed systems evolution:

1. First you monolith (co-locate everything)

2. Then you split (microservices / disaggregate)

3. Then you realize the split created new overhead

4. Then you build smart routing and hybrid approaches

We're at stage 4 now for LLM serving. The architecture is no longer "split or not" — it's "when to split, how much to split, and who decides at runtime."

What's Still Unsolved

Network bandwidth as the ceiling. KV cache transfer between prefill and decode nodes is bounded by interconnect speed. For a 128K-context 70B-parameter model, we're talking tens of gigabytes per transfer. At high request rates, the network fabric between the two pools saturates before either GPU pool does. Faster interconnect technologies (CXL, RDMA) help, but the fundamental bandwidth-vs-cache-size tension isn't resolved.

Scheduling across three (or more) node types. PPD introduced a third role. TaiChi has two instance types with configurable ratios. AMPD routes adaptively. As the number of node types grows, the scheduling problem becomes combinatorially harder — and latency-sensitive. A bad routing decision adds 50-200ms you can't take back.

Heterogeneous hardware matching. Splitwise showed you can use cheaper hardware for decode. But which cheaper hardware? The optimal decode machine depends on model size, context length, batch size, and SLO requirements. There's no standard "decode-optimized" SKU yet. The hardware market hasn't caught up to the workload split.

Agentic loop scheduling. AMPD addresses multi-round. But real agent workloads have unbounded nesting — model calls tool, tool calls sub-model, sub-model calls another tool. The prefill/decode boundary becomes fractal. How do you schedule a recursive inference graph across disaggregated pools? Nobody has a clean answer.

References

1. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., OSDI 2024) — https://arxiv.org/abs/2401.09670

2. Splitwise: Efficient Generative LLM Inference with Phase Splitting (Patel et al., ISCA 2024) — https://arxiv.org/abs/2311.18677

3. Mooncake: A KVCache-centric Disaggregated Architecture (Moonshot AI, 2024) — https://arxiv.org/abs/2407.00079

4. AMPD: Efficient Multi-round LLM Inference over Disaggregated Serving (He et al., 2026) — https://arxiv.org/abs/2602.14516

5. Not All Prefills Are Equal: PPD Disaggregation (Li et al., ICML 2026) — https://arxiv.org/abs/2603.13358

6. TaiChi: Unifying Disaggregation and Aggregation (Wang et al., 2025) — https://arxiv.org/abs/2508.01989

7. EcoInfer: Iteration-level GPU Frequency Control (Hu & Li, 2026) - https://www.mdpi.com/2079-9292/15/10/2139

8. throttLL'eM: SLO-aware GPU Frequency Scaling (2024/2025) — https://arxiv.org/abs/2408.05235