Paper: "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022) ArXiv: https://arxiv.org/abs/2203.15556 Key Insight: Previous scaling laws suggested training large models on small datasets. Chinchilla shows the opposite: allocate compute equally between model size and tokens.
The Problem: How Much to Train?
Before Chinchilla, the question was: given a compute budget, how many parameters and how many tokens?
Kaplan et al. (2020) suggested:
- N (parameters) should be much larger than D (tokens)
- E.g., 10B model on 200B tokens
Chinchilla (2022) discovered:
- N and D should be roughly equal
- E.g., 70B model on 1.4T tokens
Scaling Law Formula
Loss(N, D) = a * N^(-α) + b * D^(-β) + c
Where:
N = number of parameters
D = number of tokens
α ≈ 0.074
β ≈ 0.0617
For compute-optimal (C FLOP budget):
N ≈ C / (6 * D)
D ≈ C / (6 * N)
To maximize performance:
Increase N and D proportionally
Practical Implications
Compute Budget: 1 exaflop (10^18 FLOP)
Old Strategy (Kaplan):
- N = 100B parameters
- D = 100B tokens
- Undertrained, poor performance
Chinchilla Strategy:
- N ≈ 50B parameters
- D ≈ 1.4T tokens
- Compute-optimal, better performance
Benchmarks
Same compute budget: 1.3 exaflop
Chinchilla 70B:
- MMLU: 67.5%
- Trained on 1.4T tokens
GPT-3 175B:
- MMLU: 58.5%
- Trained on 300B tokens (undertrained)
Chinchilla 70B < GPT-3 175B in size
But Chinchilla > GPT-3 in quality!
Key: Better data allocation, not model size.
Our Analysis
Chinchilla's impact was massive. It showed that the industry was undertrained—training large models on too little data. This shifted focus to data quality and training longer. Most modern LLMs now follow Chinchilla-style scaling:
- LLaMA: Follow Chinchilla closely
- Llama-2: Scale according to Chinchilla
- Mistral: Chinchilla-optimal scaling
Code Example: Computing Optimal Allocation
import numpy as np
def compute_optimal_scaling(compute_budget_flops):
"""
Given compute budget, return optimal N and D
"""
# Scaling law constants
a = 406.4 # loss constant
alpha = 0.074
beta = 0.0617
# For compute-optimal:
# C = 6 * N * D (roughly)
# So N ≈ D ≈ sqrt(C / 6)
optimal_N = np.sqrt(compute_budget_flops / 6)
optimal_D = np.sqrt(compute_budget_flops / 6)
return optimal_N, optimal_D
# Example: 1 exaflop
budget = 1e18
N, D = compute_optimal_scaling(budget)
print(f"Optimal N: {N:.2e} parameters")
print(f"Optimal D: {D:.2e} tokens")
# Output:
# Optimal N: 4.08e+10 parameters (40B)
# Optimal D: 4.08e+13 tokens (40T)
References
- Paper: Chinchilla - Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)
- Related: Kaplan et al. Scaling Laws for Neural Language Models (2020)
Conclusion
Chinchilla established that compute allocation matters as much as total compute. This principle now guides all LLM training. Understanding scaling laws helps predict model quality and plan training budgets. Next: emergence and how new capabilities appear at scale.