Deep networks suffer from gradient vanishing: backpropagating through many layers multiplies gradients by small numbers (< 1), causing gradients to shrink exponentially. Residual connections add a shortcut: the output includes both the transformed input and the original input, preserving gradient flow.
Gradient Vanishing Problem
Loss = L(output)
dL/dweight_layer_1 = dL/doutput * ∂output/∂hidden_50 * ... * ∂hidden_2/∂weight_1
If each ∂hidden_i/∂hidden_{i-1} < 1, the product shrinks exponentially.
Residual Connections
class ResidualBlock(nn.Module):
def __init__(self, in_channels, out_channels): super().__init__() self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1) self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
def forward(self, x): residual = x out = F.relu(self.conv1(x)) out = self.conv2(out) out = out + residual # Skip connection! return F.relu(out)
The skip connection preserves the original signal, preventing gradient vanishing.
Impact
Without residual: 50-layer networks fail to train
With residual: 152-layer ResNet trains easily
Residual connections enable training of very deep networks.
Conclusion
Gradient flow is fundamental to deep learning. Residual connections solve gradient vanishing by providing shortcuts. Understanding this mechanism explains why ResNets and Transformers work and how to design deeper networks. Next: training stability and avoiding mode collapse.