Weight Initialization Strategies: Xavier, He Initialization, and Training from Scratch

Weight initialization determines whether training converges. Random uniform weights can lead to dead neurons (stuck at zero activation) or gradient vanishing (gradients too small). Xavier and He initialization set weight variance based on layer size and activation function.

Bad Initialization

# Too large: Exploding gradients
weights = torch.randn(100, 100) * 10  # Variance too high

# Too small: Vanishing gradients
weights = torch.randn(100, 100) * 0.01  # Variance too low

Xavier Initialization

For tanh activations:

# Variance = 2 / (fan_in + fan_out)
fan_in = 100
fan_out = 64
weights = torch.randn(fan_in, fan_out) * math.sqrt(2 / (fan_in + fan_out))

He Initialization

For ReLU activations:

# Variance = 2 / fan_in (ReLU kills negative half)
fan_in = 100
weights = torch.randn(fan_in, 64) * math.sqrt(2 / fan_in)

# PyTorch implements this
nn.Linear(100, 64)  # Initialized with He init by default

Conclusion

Weight initialization dramatically affects training speed and convergence. Xavier and He init match activation functions and layer sizes. Understanding initialization prevents many subtle training failures. Next: understanding gradient flow with residual connections.