Weight initialization determines whether training converges. Random uniform weights can lead to dead neurons (stuck at zero activation) or gradient vanishing (gradients too small). Xavier and He initialization set weight variance based on layer size and activation function.
Bad Initialization
# Too large: Exploding gradients
weights = torch.randn(100, 100) * 10 # Variance too high
# Too small: Vanishing gradients weights = torch.randn(100, 100) * 0.01 # Variance too low
Xavier Initialization
For tanh activations:
# Variance = 2 / (fan_in + fan_out)
fan_in = 100 fan_out = 64 weights = torch.randn(fan_in, fan_out) * math.sqrt(2 / (fan_in + fan_out))
He Initialization
For ReLU activations:
# Variance = 2 / fan_in (ReLU kills negative half)
fan_in = 100 weights = torch.randn(fan_in, 64) * math.sqrt(2 / fan_in)
# PyTorch implements this nn.Linear(100, 64) # Initialized with He init by default
Conclusion
Weight initialization dramatically affects training speed and convergence. Xavier and He init match activation functions and layer sizes. Understanding initialization prevents many subtle training failures. Next: understanding gradient flow with residual connections.