← Back
AI Mechanics

Building GPT from Scratch — Part 1: What Is GPT?

Drew Zhu·1mo ago·4 min read··👀 2652

TL;DR

We build a working GPT from scratch — tokenizer, transformer, pretraining, fine-tuning, and inference with KV cache — in ~500 lines of PyTorch on a MacBook. The model is tiny (3.3M params, trains in 1 minute) but uses the exact same architecture as GPT-4. Key takeaways: (1) BPE tokenization is just compression — frequent byte pairs merge into tokens. (2) Self-attention lets every token dynamically attend to all previous tokens; the causal mask is what makes it generative. (3) Pretraining's only objective is next-token prediction — grammar, facts, and reasoning emerge as side effects. (4) Fine-tuning changes behavior (answer questions vs. continue text) without changing the architecture or loss function. (5) KV cache turns inference from O(n²) to O(n) by caching past Keys/Values. Written for infra and software engineers who work around AI daily but want to understand what's actually inside the black box.

#buildgptfromscratch#aimechanics#gpt

If you work in infrastructure — whether that's ML platforms, serving systems, or capacity planning — you hear terms like "transformer", "attention", "KV cache", and "tokenizer" every day. You provision GPUs for them, optimize latency around them, and debug serving failures caused by them. But what do they actually do?

This series builds a complete GPT language model from scratch in ~500 lines of PyTorch, running entirely on a MacBook. Same decoder-only transformer architecture as GPT-2/3/4 — just smaller. No distributed training, no cloud GPUs, no pre-built libraries. Just PyTorch and first principles.

The goal isn't to train a production model. It's to give engineers who work around AI a first-hand understanding of what's happening inside the black box — so the next time you're debugging a CUDA OOM, tuning batch sizes for a serving fleet, or sizing KV cache memory, you know exactly what those numbers map to in the model.

The build follows the real-world order: get data → tokenize → build model → pretrain → fine-tune → run inference. All code is open-source at github.com/the-last-programmers/gpt-from-scratch.

Why Build from Scratch?

Everyone uses ChatGPT. Very few people understand what's actually happening inside — including many engineers who build the infrastructure that runs it.

As a software engineer or infra engineer, you probably know these terms at a high level: attention, embeddings, tokens, next-token prediction. But there's a gap between "I've read the Wikipedia summary" and "I understand why the KV cache saves O(n) compute per generated token." Building the thing yourself — even a tiny version — closes that gap permanently.

By the end, you'll have a working model that generates text, answers questions, and runs entirely on your laptop. More importantly, you'll understand why each piece exists and how it contributes to the final result.

Table of Contents

  1. What Is GPT

  2. Data + Tokenization

  3. The Transformer Architecture

  4. Pretraining

  5. Fine-tuning

  6. Inference

What Is GPT, Actually?

GPT stands for Generative Pre-trained Transformer. Let's break that down:

  • Generative: it produces text one token at a time, left to right

  • Pre-trained: it learns language structure from a massive text corpus before being specialized

  • Transformer: the neural network architecture it's built on (introduced in "Attention Is All You Need", 2017)

GPT is a decoder-only transformer. This means:

  • It only looks backward — each token can attend to all previous tokens, never future ones

  • It generates autoregressively — predicting one token at a time, feeding each prediction back as input

  • There is no separate "encoder" — unlike models like BERT or T5

This is the same architecture behind GPT-2, GPT-3, GPT-4, Claude, and most modern large language models.

The Full Architecture

Here's what we're building, end to end:

Each component has a specific job:

## The Build Order

We follow the real-world order — the way you'd actually build this if you sat down on a weekend:

  1. Data + Tokenization — Get text, convert it to numbers the model can process

  2. Model Architecture — Build the transformer: embeddings, attention, feed-forward blocks

  3. Pretraining — Train the model to predict the next token

  4. Fine-tuning — Teach it to follow instructions instead of just continuing text

  5. Inference — Run it: generate text with KV cache and sampling strategies

Each post pairs explanation with working code. The full implementation is ~500 lines of Python total.

What We're NOT Building

To keep this tractable and educational:

  • No distributed training (single machine, single device)

  • No Flash Attention (same math, just an optimization)

  • No RLHF (we do supervised fine-tuning only)

  • No rotary position embeddings (we use learned positional, like GPT-2)

  • No mixture of experts

These are all important at scale. But the core mechanism — the thing that makes a transformer work — doesn't need any of them.

Requirements

  • Python 3.10+

  • PyTorch 2.0+

  • A Mac with Apple Silicon (MPS backend) — or any machine with a CPU

  • ~10 minutes of patience for training

pip install torch
git clone https://github.com/the-last-programmers/gpt-from-scratch
cd gpt-from-scratch
python run.py

That's it. The entire pipeline runs end-to-end with one command.

Let's start building. Next: tokenization.

Full code: github.com/the-last-programmers/gpt-from-scratch

Comments

Loading comments...

Building GPT from Scratch — Part 1: What Is GPT? | The Last Programmers | The Last Programmers