05. Inference: When Inference Becomes Training

The Problem

Here's how every inference system works today:

Model weights get loaded once. They're read-only. A request comes in, you run a forward pass (or many, for autoregressive generation), tokens come out. The cost per request is roughly predictable. Done.

This is the assumption behind vLLM, TensorRT-LLM, Triton, every serving framework that exists. Weights are static. Inference is a pure function.

But it's kind of dumb, right?

Some questions are trivial — "what's the capital of France?" — and the model knows instantly. Some questions are hard — "prove there are infinitely many primes congruent to 3 mod 4" — and the model needs to work. With static inference, both get the same compute budget. The hard one gets a wrong answer, not because the model can't solve it, but because we didn't give it enough time.

This is like running a database where every query gets the same execution plan. A single-row primary key lookup and a 10-way join across billion-row tables — same resources, same timeout. We solved this for databases decades ago with query planners and adaptive execution. LLM serving is only now catching up.

Three Flavors of "Inference That Thinks"

Flavor 1: Generate more tokens (already in production)

The simplest version. The model writes out its reasoning — exploring hypotheses, self-correcting, verifying steps — before giving a final answer. More tokens = more compute = better answers on hard problems.

This is o1, DeepSeek-R1, Claude's extended thinking. It works. It's deployed. The infrastructure impact is manageable — you just need longer timeouts and more KV cache memory per request.

Snell et al. (2025) quantified the tradeoff rigorously on MATH benchmark with PaLM 2-S*:

• With optimal test-time compute allocation, a smaller model matches a model 14x its size in total FLOPs. But — and this is the important nuance — only on easy-to-medium problems. On the hardest problems (top difficulty quintile), no amount of extra thinking helps. The model simply doesn't have the knowledge.

• Compute-optimal allocation is 4x more efficient than naive best-of-N. Instead of sampling 64 attempts and picking the best, you can get the same accuracy with 16 — if you route compute intelligently based on difficulty.

• The routing strategy differs by difficulty: easy questions → best-of-N works fine. Medium → beam search with a process verifier wins. Hardest → save your compute, nothing helps.

The infrastructure implication that nobody talks about: you need a difficulty estimator at the gateway. Before you know the answer, you need to predict how hard the question is to decide how much compute to allocate. This is the LLM equivalent of a query optimizer, and it barely exists yet. Snell et al. approximated it using average verifier scores across a few samples — workable but not cheap.

Flavor 2: Search and verify (partially deployed)

Generate multiple reasoning paths in parallel. Score them. Return the best.

Best-of-N sampling (generate 8 answers, pick the highest-scored one) is production-ready and widely used. Full tree search (MCTS-style) is rarer because it's expensive — but papers like rStar-Math show a 7B model solving 53% of AIME with it.

"Decocted Experience" (Shen et al., April 2026) adds something clever on top of search: structured memory from past attempts.

Here's the pipeline:

1. Generate 4 trajectories per problem (only keep problems with mixed success/failure)

2. The agent distills lessons from its attempts — what worked, what failed, what patterns to avoid

3. Cluster similar lessons, keep only representative ones (intermediate compression beats keeping everything — noise removal matters)

4. At test time, retrieve relevant lessons and inject them as context before reasoning

They tested on AMC/AIME math, WebShop browsing, and SWE-bench coding tasks. The results: distilled experience consistently beats raw experience (full trajectories dumped into context). And it's complementary to thinking longer — you get gains from better context AND more reasoning.

What I find interesting from an infra angle: this is a caching layer for reasoning strategies. The model doesn't just cache KV states — it caches what worked before in a compressed, retrievable form. The storage hierarchy for inference is growing: GPU HBM for KV cache, CPU RAM for offloaded pages, SSD for prefix caches, and now a lesson store for distilled experience. More tiers, more complexity, more systems to build.

Flavor 3: Actual gradient descent during inference (research only)

This is where it gets wild. Not just "think more" — actually update the model's weights while serving a request.

qTTT (Bansal et al., 2026) — the paper that made me take this seriously.

The problem they solve: you have a 100K-token document and need to answer questions about it. Standard approach: stuff it all in KV cache. That's expensive — linear memory in context length, and attention may not focus on the right parts anyway.

qTTT's approach: run 32 gradient steps that update only the query projection weights (W_Q) using a self-supervised language modeling loss over short spans (128 tokens) sampled from the document. The K and V caches stay frozen from the initial prefill. The model absorbs the document into its weights rather than holding it all in active memory.

The results on Qwen3-4B:

• LongBench-v2: +12.6 points

• ZeroScrolls: +14.1 points

• Code comprehension: 30.8 → 43.6

And the FLOP math is what makes it compelling: 32 qTTT steps × 128 tokens = same FLOPs as generating ~8,192 thinking tokens. Same compute budget, dramatically better results on long-context tasks. Thinking tokens use the same static attention (which gets diluted over long contexts). qTTT directly fixes the attention by moving queries toward the right keys via gradient descent.

The theoretical insight is clean: as context grows, the attention score gap between the "right" token and distractors shrinks (they call it "score dilution"). The gap needs to grow as O(log T) to maintain retrieval quality. Static attention can't do this. Gradient updates on queries can.

TTRL (Khattar et al., March 2026) goes further — run RL during a user session.

The mechanism: generate N rollouts per question, majority-vote for the "correct" answer, reward rollouts that agree with the majority, run GRPO. The model improves itself between your questions.

The foundational TTRL paper (Zuo et al.) showed 211% improvement in pass@1 on AIME 2024 for Qwen-2.5-Math-7B using only unlabeled test data. And crucially: performance exceeds the model's own initial majority-vote ceiling. It's genuinely learning, not just selecting.

But Khattar's paper found the dark side: TTRL amplifies whatever tendency the model already has. If it's already safe, it becomes more safe. If it's already vulnerable to jailbreaks, it becomes more vulnerable. Majority voting locks in the dominant behavior, good or bad. And in both directions, there's a "reasoning tax" — reasoning ability degrades as a side effect.

This is a safety nightmare for anyone thinking about deploying TTRL in multi-tenant systems. One user's session could amplify harmful behaviors that then persist (or leak, depending on isolation).

What's Actually in Production vs. What's Research

I want to be honest about this because too many AI infra posts present everything as if it's shipping next quarter:

The gap for Flavor 3 is not "we need better algorithms." It's "the entire serving stack assumes read-only weights." vLLM, TensorRT-LLM, SGLang — none support writing to model parameters during a request. You'd need:

• Per-request optimizer state (Adam momentum/variance buffers) in GPU memory

• Copy-on-write semantics for model weights (so one user's updates don't affect another's)

• Rollback after the request completes

• Memory accounting that includes gradient buffers in the capacity planner

It's like your web server needing to hot-patch its own code differently per incoming connection, then rollback after responding. Technically possible. Architecturally, a completely different system than what exists.

What's Still Unsolved

The difficulty estimation problem. You need to decide how much compute to spend before solving the problem. Snell et al. used verifier scores across a few samples — but that itself costs compute. It's circular. A cheap, accurate difficulty predictor would change the economics entirely.

Billing. If one request uses 50 tokens of thinking and another uses 50,000 — who pays 1000× more? How do you expose this? How do you set quotas? OpenAI charges per-token but the variance is enormous. No consensus on how to handle this fairly.

The hard problem cliff. Test-time compute only helps on problems the model could solve with enough attempts. On genuinely hard problems (top difficulty bin), more compute is just waste. But you can't know which bin you're in without trying. The optimal strategy is to try a little, estimate difficulty, and either commit more compute or give up. Nobody has a good implementation of this "progressive deepening" pattern in production.

Safety under self-modification. If TTRL amplifies existing tendencies, deploying it in a multi-tenant system where different users have different session histories could create divergent model behaviors. Alignment that's stable under further optimization is an open research problem — TTRL makes it an operational concern.

References

1. Scaling LLM Test-Time Compute Optimally (Snell et al., 2025) — https://arxiv.org/abs/2408.03314

2. Let's (not) just put things in Context: qTTT (Bansal et al., 2026) — https://arxiv.org/abs/2512.13898

3. Amplification Effects in Test-Time RL (Khattar et al., 2026) — https://arxiv.org/abs/2603.15417

4. TTRL: Test-Time Reinforcement Learning (Zuo et al., 2025) — https://arxiv.org/abs/2504.16084

5. Decocted Experience Improves Test-Time Inference (Shen et al., 2026) — https://arxiv.org/abs/2604.04373

6. DeepSeek-R1 (2025) — https://arxiv.org/abs/2501.12948

7. rStar-Math (Microsoft, 2025) — https://arxiv.org/abs/2501.04519