04. Inference: Why Serving LLMs Is Nothing Like Serving Traditional ML

The Problem

If you've served traditional ML models — image classifiers, recommendation models, fraud detectors — you know the playbook:

1. Request comes in with a fixed-size input (an image, a feature vector)

2. Model runs a single forward pass (a few milliseconds)

3. Response goes out

4. All state is discarded. Next request starts fresh.

Scale this with replicas, load balancers, and autoscaling. Straightforward. The model is a pure function — same input, same compute, same memory, every time.

LLM serving breaks every one of these assumptions:

It's sequential, not parallel. Generating 500 tokens means 500 serial forward passes. Each token depends on all previous ones. You can't parallelize the output of a single request. A 500-token response at 50ms per token takes 25 seconds — not because your system is slow, but because that's the physics of autoregressive generation.

It's stateful, not stateless. Between each token, the model stores a "KV cache" — the compressed representation of everything it's seen so far. For a 70B model with 128K context, this cache can hit 30-40GB per request. That state must persist for the duration of the generation, which could be seconds to minutes.

Memory is the bottleneck, not compute. During the decode phase (generating tokens one at a time), the GPU spends most of its time reading memory, not doing math. The model weights are read once per token. The KV cache is read once per token. The actual arithmetic (matrix multiplies) completes faster than the memory can feed it data. Your expensive GPU's tensor cores are mostly idle, waiting for HBM bandwidth.

Requests have wildly different costs. A "what's 2+2?" query touches maybe 50 tokens. A "summarize this 100-page document" query might consume 100K input tokens and generate 2K output tokens. The memory footprint, compute time, and latency differ by 1000x — but they arrive at the same endpoint.

This isn't like serving images where every request costs roughly the same. It's like running a database where one query scans 10 rows and the next scans 10 billion — and you need to guarantee latency SLOs for both.

Why This Breaks Traditional Serving Patterns

Batching doesn't work the way you think

In traditional ML: batch 64 requests together, run one big matmul, return 64 results. Every request takes the same time. Beautiful.

In LLM serving: requests arrive at different times, have different input lengths, and will generate different output lengths. If you batch a 10-token generation with a 5000-token generation, the short request finishes in 200ms but sits around waiting for the long one to complete before the batch releases.

The fix: continuous batching (Orca, 2022). Requests join and leave the batch on every iteration, not per-batch. It's conceptually similar to how modern event loops handle connections — you don't block on one slow connection, you multiplex.

Caching is essential but the eviction policy is hard

The KV cache is the dominant memory consumer. A busy 8-GPU node serving a 70B model might have 640GB of HBM total, with the model weights taking 140GB and KV caches consuming most of the rest. How you allocate, fragment, and evict those caches determines how many concurrent requests you can serve.

This is where vLLM's PagedAttention made its mark.

Load balancing can't just round-robin

A request with 100K input tokens and 1K output will consume 50x more GPU memory and 20x more time than a short chat message. Round-robin routing means one replica gets crushed while another is idle. You need request-aware routing — which means your load balancer needs to understand the model's memory accounting. That's a level of coupling between routing and runtime that most serving infra deliberately avoids.

The Technical Transformation

PagedAttention: "What if we managed KV cache like OS virtual memory?"

Before vLLM (2023), serving systems pre-allocated KV cache memory contiguously for each request — reserving worst-case space (max sequence length) upfront. If max context is 128K tokens but the actual request uses 2K, you've wasted 98% of the reservation. Across a batch of 64 requests, the fragmentation is catastrophic. Real systems saw 60-80% of GPU memory wasted on internal fragmentation.

vLLM's insight: this is the same problem OS virtual memory solved in the 1960s.

• Split KV cache into fixed-size pages (blocks of 16 tokens)

• Maintain a page table that maps logical token positions to physical memory blocks

• Allocate pages on demand as tokens are generated — no pre-reservation

• Pages can be non-contiguous in physical memory — just like OS pages

Result: near-zero fragmentation. 2-4x more requests can fit in the same GPU memory. This single idea changed the economics of LLM serving overnight.

If you've ever debugged memory fragmentation in a long-running allocator (malloc, jemalloc) or worked with memory-mapped files — this is the same problem and the same class of solution, just on GPU HBM instead of system RAM.

MLA: "What if we compressed the KV cache at the architecture level?"

PagedAttention eliminates fragmentation. But the cache is still large. For very long contexts (128K+) even perfectly managed KV cache eats enormous memory.

DeepSeek-V3 attacked this differently: Multi-Head Latent Attention (MLA). Instead of storing full key-value vectors per layer per token (the standard approach), MLA compresses them into a low-dimensional latent vector through a learned projection.

The numbers: MLA reduces KV cache memory to roughly 1/6th of standard Grouped-Query Attention (GQA). For a 128K context window, that's the difference between "we need 8 GPUs per request" and "we can handle it on 2."

The catch: you had to train with MLA from scratch. Or so everyone thought.

MHA2MLA (2025): "Actually, you can retrofit it"

This paper showed you can convert existing GQA/MHA models to MLA format with only 0.6-1% of the original training data. Over 92% KV cache reduction, negligible quality loss. No retraining from scratch.

From an ops perspective: this means your existing fleet of Llama/Mistral deployments can potentially get a massive memory efficiency upgrade through a cheap fine-tuning step. That's rare — usually architectural improvements require retraining the whole model.

The Storage Hierarchy Problem (2026 frontier)

Here's where it gets interesting. Even with PagedAttention and MLA, production systems hit a wall: GPU HBM is finite, and long-context workloads (128K-1M tokens) generate KV caches that simply don't fit.

The industry is solving this the same way CPUs solved the memory wall decades ago: tiered storage with intelligent caching policies.

 Tier 0: GPU HBM (fast, expensive, small)
↕ hot/warm cache management
Tier 1: Host CPU RAM (medium speed, larger)
↕ offloading/prefetching
Tier 2: NVMe SSD (slow, huge, cheap)

Two recent papers tackle different points in this design space:

IceCache (2026): Smart offloading to CPU RAM

The naive approach to KV cache offloading is per-page eviction — move cold pages to CPU when GPU HBM fills up. But this creates a precision problem: random token-level eviction breaks the semantic coherence of attention, degrading quality on long-context tasks.

IceCache's fix: cluster semantically similar tokens into the same pages before offloading. Tokens that the model attends to together stay together. When a block is offloaded, it's a coherent semantic unit — not random fragments. Prefetching decisions become more predictable because semantic locality implies access locality.

It's the same insight behind huge pages and THP in Linux — grouping related memory reduces TLB misses. Here, grouping related tokens reduces quality loss during offloading.

Tutti (2026): GPU-to-SSD direct path, bypassing the CPU entirely

For truly massive prefix caches (shared system prompts, document contexts cached across requests), even CPU RAM isn't big enough. Tutti puts KV cache on NVMe SSDs — but with a twist.

The problem with SSD-backed KV cache: PagedAttention's fine-grained block layout creates terrible random I/O patterns on SSDs. 16-token pages scattered across an SSD means millions of small random reads — exactly the workload SSDs handle worst.

Tutti's solution:

• GPU-native object abstractions that batch KV blocks into SSD-friendly large sequential reads

• Direct P2P (peer-to-peer) memory mapping: GPU reads straight from NVMe via PCIe, completely bypassing CPU involvement

• Converts the random-read problem into a sequential-prefetch problem

If you've worked with direct I/O, io_uring, or SPDK — this is the same philosophy. Remove the kernel from the data path. Let the hardware talk directly.

The Revenue Angle

As model companies like Anthropic and OpenAI push into enterprise, their revenue is surging — and inference is the gas that fuels the money machine. Every point of serving efficiency translates directly to margin.

The math is stark: if your inference cost per token drops 50% through better KV cache management, you either double your margin or halve your prices and grow volume. At the scale these companies operate (billions of tokens per day), even a 10% efficiency gain is worth tens of millions annually.

Inference isn't a cost center anymore. It's the revenue engine. And every paper above — PagedAttention, MLA, IceCache, Tutti — is directly attacking the bottleneck that determines how much money each GPU generates per hour.

What's Still Unsolved

• The scheduling-memory co-optimization problem. Deciding which requests to batch, how much memory to reserve, and when to preempt — these decisions are tightly coupled, and current systems still solve them somewhat independently. It's like optimizing a database query plan while simultaneously managing the buffer pool, without the two systems talking to each other.

• KV cache sharing across requests. Many requests share prefixes (same system prompt, same document). In theory you can share those KV cache pages. In practice, coordinating shared pages across requests with different lifetimes, across multiple GPUs, without corrupting state — it's reference counting and GC in a real-time system. Fragile.

• The long-context quality gap. Offloading and compression work great on benchmarks, but production users report quality degradation on "needle-in-a-haystack" tasks when aggressive eviction or compression is applied. The models lean on caches in ways benchmarks don't test.

• Cost of switching. Moving from vLLM to a system with MLA, or from HBM-only to tiered storage, isn't a config change. It's a re-architecture. The migration path for production systems is unclear.

Of all the layers in AI Infra, inference is the one I find most interesting — and the most consequential. It's the actual money machine, the day-to-day power burner, the thing running 24/7 serving real users. I'll be diving deeper into each of the papers and aspects covered here in the upcoming posts.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) — https://arxiv.org/abs/2309.06180

2. DeepSeek-V3 Technical Report / MLA (DeepSeek-AI, 2024) — https://arxiv.org/abs/2412.19437

3. MHA2MLA: Towards Economical Inference (Ji et al., 2025) — https://arxiv.org/abs/2502.14837

4. IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs (Mao et al., 2026) — https://arxiv.org/abs/2604.10539

5. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving (Qiu et al., 2026) — https://arxiv.org/abs/2605.03375

6. Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., 2022)