Optimizers are algorithms that update weights based on gradients. SGD is simple but slow. Momentum accelerates convergence. Adam adapts learning rates per parameter. Each optimizer has strengths: SGD generalizes well, Adam is fast, RMSprop for non-stationary problems.
SGD (Stochastic Gradient Descent)
# Update: w = w - lr * gradient
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
Pros: Simple, generalizes well. Cons: Slow convergence, sensitive to learning rate.
Momentum
# Momentum accumulates gradients: v = β*v + gradient
# Update: w = w - lr * v optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
Pros: Faster convergence, escapes local minima. Cons: Can overshoot.
Adam (Adaptive Moment Estimation)
# Adapts learning rate per parameter
# m = β1*m + (1-β1)*gradient # v = β2*v + (1-β2)*gradient^2 # Update: w = w - (lr * m) / (sqrt(v) + ε) optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
Pros: Fast, automatic learning rate tuning. Cons: Can overfit, memory overhead.
When to Use
- SGD: Simple models, generalization matters
- Momentum: CNNs, need faster convergence
- Adam: LLMs, complex objectives, most practical choice
- RMSprop: Non-stationary data, sparse gradients
Conclusion
Choosing the right optimizer affects training speed and final performance. Adam is the practical default; SGD for maximum generalization. Understanding optimizer mechanics guides hyperparameter tuning and model development. Next: learning rate scheduling—how to adapt learning rate during training.