All modern LLMs are Transformers: neural networks with self-attention. From GPT-1's decoder-only stack to GPT-4's multi-expert mixture, the core building blocks remain the same. Understanding attention, positional encoding, and scaling laws explains why Transformers scale better than RNNs and CNNs.
Self-Attention Mechanism
Self-attention computes relationships between all tokens: how much should each token attend to every other token?
import torch
import torch.nn.functional as F
def scaled_dot_product_attention(Q, K, V, mask=None): """ Compute self-attention. Q, K, V: (batch, seq_len, d_model) """ d_k = Q.shape[-1]
# Compute attention scores scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
# Apply mask (to prevent attending to future tokens) if mask is not None: scores = scores.masked_fill(mask == 0, -1e9)
# Softmax attention_weights = F.softmax(scores, dim=-1)
# Apply to values output = torch.matmul(attention_weights, V) return output, attention_weights
Self-attention's parallelizability (vs. RNNs' sequential nature) enabled the scaling that made LLMs possible.
Conclusion
Transformers and attention are the foundation of modern LLMs. Understanding the mechanism explains why LLMs work and scales so well. The evolution from attention-is-all-you-need to modern LLMs has been about making attention more efficient and adding scale. Next: we'll explore training methodologies—pretraining, SFT, and RLHF.