Phi-2 (2.7B parameters) outperforms Llama-2 13B on code and math. Phi-3 (14B) matches 70B models on benchmarks. The secret: training data quality > model size. Phi models use synthetic data (generated by stronger models) and rigorous filtering. This challenges the assumption that "bigger = better."
Phi Training Strategy
Code Snippet
1. Synthetic Data Generation
- Use GPT-4 to generate high-quality training examples
- Reason through code problems step-by-step
- Curate examples that teach reasoning patterns
2. Data Quality Filtering
- Remove low-quality examples
- Keep only examples that improve downstream performance
- Focus on code, math, reasoning (high-value domains)
3. Smart Training
- Optimize learning rate schedule
- Use smaller models but train longer
- Focus on data quality over scale
Benchmark Performance
Code Snippet
Model | Size | MMLU | HumanEval | Math
Phi-2 | 2.7B | 56.3% | 48% | 12%
Llama-2 7B | 7B | 45.3% | 12% | 6%
Llama-2 13B | 13B | 54.8% | 30% | 10%
Phi-3 | 14B | 69.8% | 76% | 30%
Llama-2 70B | 70B | 69.2% | 73% | 25%
Phi-3 14B ≈ Llama-2 70B on reasoning tasks, 5× smaller.
Why Smaller Can Be Better
Code Snippet
# Training compute budget: FLOP budget = model_size * tokens
# Fixed budget: choose between:
# A) Large model, fewer tokens (70B × 1T tokens)
# B) Small model, more tokens (14B × 5T tokens)
# Phi strategy: Option B
# - More data coverage
# - Better memorization of reasoning patterns
# - Faster inference
# - Easier to fine-tune
Conclusion
Phi demonstrates that data quality and careful training beats raw scale. Small models with excellent training data are more practical than large models trained on quantity. This is the future of LLMs: optimized, focused, efficient. Next: Mixtral—mixture of experts and dynamic routing.