Regularization prevents overfitting: models fitting noise rather than signal. L1/L2 penalize large weights. Dropout randomly disables neurons. Early stopping halts when validation loss stops improving.
L1/L2 Regularization
Code Snippet
# L2 (ridge): loss += λ * ||w||^2
# Shrinks weights toward zero
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-5)
# weight_decay = L2 regularization strength
# L1 (lasso): loss += λ * ||w||
# Drives weights to exactly zero (feature selection)
Dropout
Code Snippet
class RegularizedModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(100, 64)
self.dropout = nn.Dropout(p=0.5) # Drop 50% of neurons
self.fc2 = nn.Linear(64, 10)
def forward(self, x):
x = F.relu(self.fc1(x))
x = self.dropout(x) # Random neurons disabled
x = self.fc2(x)
return x
Dropout forces the network to learn redundant representations.
Early Stopping
Code Snippet
best_val_loss = float('inf')
patience = 10
patience_counter = 0
for epoch in range(num_epochs):
train_loss = train()
val_loss = validate()
if val_loss < best_val_loss:
best_val_loss = val_loss
patience_counter = 0
torch.save(model.state_dict(), 'best_model.pth')
else:
patience_counter += 1
if patience_counter >= patience:
print("Stopping early")
break
Conclusion
Regularization prevents overfitting by constraining model complexity. L2 shrinks weights, dropout learns redundancy, early stopping stops before overfitting. Combining techniques is more effective than any single technique alone. Next: batch normalization and layer normalization.