Gradient Flow and Residual Connections: Skip Connections Enable Deep Networks

Deep networks suffer from gradient vanishing: backpropagating through many layers multiplies gradients by small numbers (< 1), causing gradients to shrink exponentially. Residual connections add a shortcut: the output includes both the transformed input and the original input, preserving gradient flow.

Gradient Vanishing Problem

Loss = L(output)
dL/dweight_layer_1 = dL/doutput * ∂output/∂hidden_50 * ... * ∂hidden_2/∂weight_1

If each ∂hidden_i/∂hidden_{i-1} < 1, the product shrinks exponentially.

Residual Connections

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
    
    def forward(self, x):
        residual = x
        out = F.relu(self.conv1(x))
        out = self.conv2(out)
        out = out + residual  # Skip connection!
        return F.relu(out)

The skip connection preserves the original signal, preventing gradient vanishing.

Impact

Without residual: 50-layer networks fail to train
With residual: 152-layer ResNet trains easily

Residual connections enable training of very deep networks.

Conclusion

Gradient flow is fundamental to deep learning. Residual connections solve gradient vanishing by providing shortcuts. Understanding this mechanism explains why ResNets and Transformers work and how to design deeper networks. Next: training stability and avoiding mode collapse.