Building GPT from Scratch — Part 5: Fine-tuning

The Problem

After pretraining, our GPT can produce English text. But it has a fundamental limitation: it doesn't know what you want it to do. Ask it a question, and it might continue the question with more questions. Give it an instruction, and it might write an article about instructions.

The model has learned language. It hasn't learned behavior.

Consider the difference:

Prompt: "What is the capital of France?"

Pretrained model output:
"What is the capital of France? What is the capital of Germany?
What is the capital of Spain?..."
(It learned that questions often come in lists)

Fine-tuned model output:
"The capital of France is Paris."
(It learned that questions should be answered)

Same model, same architecture, same parameters (mostly). The difference is what data we trained it on last.

What Fine-tuning Actually Does

Fine-tuning does NOT teach the model new knowledge. The knowledge comes from pretraining (it saw the answer "Paris" thousands of times in the training corpus).

Fine-tuning teaches the model a new format: when you see a question/instruction, produce an answer/response. It's behavioral alignment, not knowledge injection.

The mechanism is identical to pretraining:

• Same loss function (cross-entropy, next-token prediction)

• Same optimizer (AdamW)

• Same forward pass

The only difference is the data format.

The Data Format

Pretraining data (unstructured text):

The cat sat on the mat. It was a warm sunny day and
the birds were singing in the trees outside...

Fine-tuning data (structured instruction-response pairs):

### Instruction:
What is gravity?

### Response:
Gravity is the force that attracts objects with mass toward each other.

The model learns: when I encounter the pattern ### Instruction:\n...\n### Response:\n, what follows should be a direct answer. The special markers (### Instruction:, ### Response:) act as format tokens that signal the expected behavior.

This is why ChatGPT responds to questions instead of continuing them — it was fine-tuned on millions of (question, answer) pairs.

The Critical Implementation Detail: Masking Padding

Instruction-response samples vary wildly in length:

Sample 1: "What is 2+2?" → "4."                    (12 tokens)
Sample 2: "Explain photosynthesis." → "Plants..."    (85 tokens)

But the model needs fixed-length sequences for batched training. We pad short sequences:

token_ids = tokenizer.encode(prompt)
if len(token_ids) <= context_length:
    seq_len = len(token_ids)
    # Pad to context_length with zeros
    token_ids = token_ids + [0] * (context_length - seq_len)

The problem: if we compute loss on the padding tokens, the model wastes capacity learning to predict zeros — which is useless and actively harmful.

The solution: mask the loss on padding positions:

PAD_TOKEN = -100  # PyTorch's special ignore value

def __getitem__(self, idx):
    tokens, seq_len = self.samples[idx]
    x = tokens[:-1]
    y = tokens[1:].clone()

    # Tell cross-entropy to IGNORE these positions
    y[seq_len - 1:] = PAD_TOKEN
    return x, y

And in the loss computation:

loss = F.cross_entropy(
    logits.view(-1, logits.size(-1)),
    targets.view(-1),
    ignore_index=-100,  # ← skip positions where target is -100
)

PyTorch's cross_entropy with ignore_index=-100 computes loss ONLY on real tokens. Padding contributes zero to the gradient. This seems like a minor detail, but getting it wrong is a common bug that silently degrades model quality.

Differences from Pretraining

Lower learning rate: the model already has good representations from pretraining. Large updates would destroy that knowledge. We want small, precise adjustments.

Lower weight decay: weight decay pushes parameters toward zero (regularization). During fine-tuning, we don't want to regularize away what pretraining learned — we want to preserve it while adding new behavior on top.

Fewer steps: with only 20 training samples, overfitting happens fast. After 500 steps, the model has seen each sample ~100 times. More training would just memorize the exact responses.

The Fine-tuning Loop

model.train()
for step in range(max_steps):
    x, y = get_batch(train_dataset, batch_size, device)

    # Forward pass
    logits, _ = model(x)

    # Loss only on non-padded positions
    loss = F.cross_entropy(
        logits.view(-1, logits.size(-1)),
        y.view(-1),
        ignore_index=-100,
    )

    # Standard backward pass
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    optimizer.zero_grad()

Structurally identical to pretraining. The model doesn't "know" it's being fine-tuned — it's still just minimizing next-token prediction loss on whatever data you give it.

What Changes in the Model

Fine-tuning modifies ALL parameters. But the changes are small — the weights shift slightly from their pretrained values. Conceptually:

Pretrained weights:  W = [learned language structure]
Fine-tuned weights:  W' = W + δ  (small perturbation)

The model retains its language ability and gains instruction-following behavior. This is why fine-tuning needs so little data compared to pretraining — it's making a small adjustment, not learning from scratch.

Advanced: LoRA (Low-Rank Adaptation)

For larger models, fine-tuning all parameters is expensive. LoRA freezes the original weights and trains small "adapter" matrices:

Original: y = Wx
LoRA:     y = Wx + BAx  (where B and A are small matrices)

W is frozen (175B parameters → no gradients, no optimizer state)
B×A is trained (maybe 1M parameters → cheap)

We don't use LoRA in our tiny model (3.3M params is already cheap to fine-tune fully). But it's how GPT-4, Claude, and other production models are adapted to specific tasks without retraining all parameters.

Results

Before fine-tuning:

Prompt: "### Instruction:\nWhat is gravity?\n\n### Response:\n"
Output: "### Instruction:\nWhat is gravity?\n\n### Response:\nand the
         bright stars which would of a little of the creature..."

After fine-tuning:

Prompt: "### Instruction:\nWhat is gravity?\n\n### Response:\n"
Output: "### Instruction:\nWhat is gravity?\n\n### Response:\nGravity
         is the force that attr..." (truncated)

The quality is limited by our model's tiny size — 3.3M parameters can't store much factual knowledge. But the format is correct: it generates a response instead of continuing randomly. The mechanism works.

At GPT-3/4 scale (100B+ parameters), this same technique produces the fluent, knowledgeable assistants you interact with daily.

The Full Pipeline So Far

Raw text                     → BPE Tokenizer    → Token IDs
Token IDs + Random Weights   → Pretraining      → Language Model
Language Model + Instruction Data → Fine-tuning  → Instruction-Following Model

Three stages, each building on the last. The architecture never changes. Only the data and the objective's implicit goal change:

1. Tokenizer: learn compression

2. Pretraining: learn language

3. Fine-tuning: learn behavior

Key Takeaways

1. Fine-tuning is format training, not knowledge training. The model already "knows" facts from pretraining. Fine-tuning teaches it when and how to express them.

2. The loss function is identical. There's no special "instruction loss" or "alignment loss." It's still cross-entropy next-token prediction. The structure of the data does all the work.

3. Mask your padding. If you don't ignore padded positions in the loss, the model learns to generate padding — a subtle but impactful bug.

4. Less is more. Lower learning rate, lower regularization, fewer steps. Fine-tuning is a delicate adjustment, not a heavy training run.

5. This is how ChatGPT was made. Pretrain on the internet (expensive, done once) → fine-tune on conversations (cheap, done repeatedly with new data). The same pipeline at different scales.

Full code: github.com/the-last-programmers/gpt-from-scratch

Next: inference — how to actually run the model and generate text efficiently.