03. Post-training: RLHF Is Dead, Long Live RL

The Problem

After pretraining, your model knows a lot but behaves badly. It hallucinates, ignores instructions, produces harmful content, and can't reason step-by-step. Post-training fixes this.

The original approach (2022-2023 era):

Collect human preference data — "output A is better than output B"
Train a reward model on those preferences
Run PPO (Proximal Policy Optimization) against the reward model

It worked. ChatGPT shipped on this. But it had ugly infrastructure requirements:

You needed four models running simultaneously — the policy (actor), the reward model, a reference model (for KL penalty), and a critic (value function). On a 70B model, that's 4x the GPU memory. You needed humans in the loop generating preference data constantly. And PPO is notoriously unstable — hyperparameter-sensitive, prone to reward hacking, and expensive to debug at scale.

It's like running a distributed transaction across four coupled services where any one of them drifting causes the whole system to produce garbage. Debugging is a nightmare because the failure mode isn't a crash — it's subtle quality degradation that shows up days later in eval metrics.

What Changed: The DeepSeek-R1 Moment

DeepSeek-R1 (January 2025) was a turning point. Not because it matched o1 on math benchmarks — but because of how it got there.

They trained DeepSeek-R1-Zero: pure RL on a base model. No SFT warmup. No human-labeled reasoning traces. Just: "here's a math problem, here's whether your answer was right, optimize."

And something unexpected happened around step 8,000 of training.

The model started writing "Wait." in its reasoning chains. Then "Wait, let me reconsider." Then full self-correction loops — catching its own errors and restarting from a better point. They called it the "aha moment." The model invented self-reflection purely from reward pressure.

This matters for infra because it redefines what post-training IS. It's no longer "fine-tune on curated data." It's "let the model explore, generate thousands of reasoning trajectories, score them, and learn from the distribution." The infrastructure looks less like a training job and more like a search system with a training loop wrapped around it.

The key innovation: GRPO

Group Relative Policy Optimization eliminates the critic model entirely. Instead of training a separate value function to estimate "how good is this state" (the way PPO does), GRPO just:

Samples a group of outputs for each prompt
Scores them all
Normalizes advantages within the group: A_i = (r_i - mean) / std

That's it. No value model. No GAE. The "baseline" is just the group average.

From an infra standpoint: you just killed one of the four models. The tradeoff is you need more samples per prompt (typically 8-64 rollouts per question). So you traded model-parallelism complexity for generation throughput requirements. Whether that's a good trade depends entirely on how fast your inference engine is.

The Trend: Where Post-Training Is Going

DeepSeek-R1 opened a door. The papers that followed walked through it:

Re² (Chen et al., 2026) — trains models to know when to give up on a reasoning path and restart. Pure RL, no SFT. The model learns to estimate: "what's my probability of success if I continue this chain vs. starting fresh?" It's like teaching a search algorithm to prune bad branches earlier — except the algorithm is the model itself, and it learns the pruning policy from experience.

INTUITOR (Zhao et al., 2025) — solves the "but what about domains without verifiers?" problem. Math has clear right/wrong answers. Creative writing doesn't. INTUITOR uses the model's own internal confidence as a reward signal. The model learns to trust its uncertainty estimates and optimize toward reasoning paths where it's more certain. Sketchy? Maybe. But it extends RL post-training to domains where you can't just check an answer key.

Latent-GRPO (2026) — this one is interesting from an infra angle. Explicit chain-of-thought is expensive. A model reasoning through a hard problem might generate 10K tokens of "thinking" before producing a 50-token answer. That's 10K tokens of KV cache, 10K decode steps, 10K tokens of memory bandwidth. Latent-GRPO compresses that reasoning into continuous latent vectors — 3-4x shorter generation chains with equivalent accuracy. Your inference cost per reasoning query just dropped 70%.

The trajectory is clear: post-training is becoming pure RL → self-improvement → compression. Less human involvement at each step. More compute at each step. And the infra requirements shift accordingly.

The Infrastructure Reality

Here's what actually runs during a modern RL post-training job:

80% of your compute is generation. Not training. Generation.

This flips everything. In pretraining, your GPUs are doing matmuls on training batches — compute-bound, predictable, steady-state. In RL post-training, your GPUs are running inference to generate rollouts — memory-bound, variable-length, bursty.

It's like discovering that your "batch processing pipeline" is actually 80% serving real-time requests to itself. The performance characteristics are completely different. You need inference optimization (KV cache management, continuous batching, speculative decoding) inside your training loop.

The resharding problem

Here's the part that makes infra engineers twitch:

During generation, you want the model sharded for inference efficiency — tensor parallelism across a few GPUs, high throughput, PagedAttention on the KV cache.

During the gradient update, you want it sharded for training efficiency — FSDP or 3D parallelism, large batch aggregation, optimizer states distributed across many GPUs.

These are different optimal configurations. So every iteration, you're resharding the model between two parallelism strategies. Frameworks like veRL (HybridFlow) and OpenRLHF are built specifically to handle this transition — but it's still expensive. It's like having a database that needs to switch between OLTP and OLAP layouts every 30 seconds.

DeepSeek-R1's actual setup

512 H800 GPUs (64 nodes × 8 GPUs)
4 components: Rollout workers (vLLM), Inference workers (reward/reference models), Rule-based reward, Training workers
Total training cost for R1: ~$294K in GPU hours
R1-Zero trained in ~198 hours
The full R1 pipeline (with SFT stages): ~4 days

Not cheap, but remarkably efficient for a model that matches o1. The cost is dominated by generation — sampling 8-64 rollouts per prompt, across hundreds of thousands of prompts, repeated over thousands of RL steps.

Why This Matters If You're Building Infra

Your post-training cluster is mostly an inference cluster. Size the generation pool for throughput. The training component is the minority workload. Budget GPUs accordingly.
Variable-length generation is your scheduling nightmare. Unlike pretraining where every batch is the same shape, RL rollouts produce outputs from 50 to 50,000 tokens. Your scheduler needs to handle this variance without wasting memory or starving short requests.
The reward function is your reliability boundary. If the reward signal is noisy, unstable, or hackable — the model will exploit it. This isn't a training bug, it's the system working as designed. Build monitoring for reward distribution drift the same way you'd monitor for metric cardinality explosion in a telemetry pipeline.
Latent reasoning compression is coming for your inference costs. If Latent-GRPO style approaches work broadly, the "long thinking" tax on inference serving drops dramatically. Plan for a future where reasoning models are 3-4x cheaper to serve per query than they are today.

What's Still Unsolved

Reward signal for open-ended tasks. Math and code have verifiers. "Write a good email" doesn't. INTUITOR is a start but self-reward risks collapse (model rewards itself for confidently wrong answers).
Generation throughput at RL scale. 80% of the loop is generation, and you need 8-64 samples per prompt. This is millions of inference calls per training run. Existing serving systems aren't designed to be embedded inside training loops at this scale.
When to retrain vs. when to keep exploring. RL can overfit to the reward distribution. Knowing when to stop — or when to switch reward functions — is still an art, not a science.
Distillation gap. DeepSeek distilled R1's reasoning into smaller models (1.5B-70B). The quality drop is real. Whether latent compression (Latent-GRPO) closes this gap or just hides it is an open question.

References

DeepSeek-R1: Incentivizing Reasoning Capability via Reinforcement Learning (DeepSeek-AI, 2025) — https://arxiv.org/abs/2501.12948
Re²: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (Chen et al., 2026) — https://arxiv.org/abs/2603.07197
Learning to Reason without External Rewards / INTUITOR (Zhao et al., 2025) — https://arxiv.org/abs/2505.19590
Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning (2026) — https://arxiv.org/abs/2604.27998
OpenRLHF — https://github.com/OpenRLHF/OpenRLHF
veRL / HybridFlow (2024) — https://arxiv.org/abs/2409.19256