Building a Large Language Model from Scratch

The language-modeling objective

A causal language model learns to predict the next token from previous tokens. Training examples are produced by shifting token sequences: the input contains a prefix and the target contains the following token at each position.

Tokens, embeddings and position

A tokenizer maps text to integer IDs. An embedding table turns each ID into a learned vector, while positional information lets the network distinguish token order. The context window defines how many tokens can influence a prediction in one pass.

Causal self-attention

Each token produces query, key and value vectors. Query–key similarity creates attention scores, and a causal mask prevents a position from reading future tokens. Normalized scores weight the value vectors to produce context-aware representations.

scores = (Q @ K.transpose(-2, -1)) / sqrt(head_dim)
scores = scores.masked_fill(causal_mask, -inf)
context = softmax(scores, dim=-1) @ V

The Transformer block

• Multi-head attention learns several relationship patterns in parallel.
• A feed-forward network transforms each position after attention.
• Residual connections and normalization stabilize deeper networks.
• The output head maps hidden states back to token probabilities.

Train small, measure carefully

A compact educational model is useful for understanding the system, but it should not be compared with large production models without controlling data, compute and evaluation. Track training and validation loss, inspect generated samples and record the complete configuration for reproducibility.