Batch normalization normalizes layer inputs to zero mean/unit variance, stabilizing training and enabling higher learning rates. Layer normalization normalizes across features (instead of batch), solving the batch-size dependency problem. Understanding their trade-offs guides architecture design.
Batch Normalization
Code Snippet
# During training: normalize by batch statistics
# During inference: normalize by running statistics (computed during training)
class BatchNormModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(100, 64)
self.bn1 = nn.BatchNorm1d(64) # Normalize across batch dimension
self.fc2 = nn.Linear(64, 10)
def forward(self, x):
x = self.fc1(x)
x = self.bn1(x) # Normalize
x = F.relu(x)
x = self.fc2(x)
return x
Batch norm reduces internal covariate shift, enabling faster training.
Layer Normalization
Code Snippet
# Normalize across features (not batch)
class LayerNormModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(100, 64)
self.ln1 = nn.LayerNorm(64) # Normalize across feature dimension
self.fc2 = nn.Linear(64, 10)
def forward(self, x):
x = self.fc1(x)
x = self.ln1(x) # Normalize
x = F.relu(x)
x = self.fc2(x)
return x
Layer norm doesn't depend on batch size, making it ideal for Transformers and variable-length sequences.
Conclusion
Batch norm and layer norm both stabilize training but have different properties. Batch norm is standard for CNNs; layer norm is standard for Transformers. Understanding when to use each informs architecture choices. Next: weight initialization and their effects on training.