Hooks and Model Surgery: Forward Hooks, Backward Hooks, and Feature Extraction Patterns

Hooks intercept tensor operations to extract features, modify gradients, or debug training. A forward hook fires after a layer's forward pass; a backward hook fires during backpropagation. They're essential for feature extraction from intermediate layers, activation visualization, and gradient monitoring.

Forward Hooks for Feature Extraction

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 32),
    nn.ReLU(),
    nn.Linear(32, 10)
)

activations = {}

def hook_fn(module, input, output):
    activations[module] = output.detach()

# Register hook on ReLU
model[1].register_forward_hook(hook_fn)

x = torch.randn(4, 10)
output = model(x)

print(f"ReLU activation shape: {activations[model[1]].shape}")
print(f"ReLU activation stats: mean={activations[model[1]].mean():.4f}, std={activations[model[1]].std():.4f}")

Output:

ReLU activation shape: torch.Size([4, 32])
ReLU activation stats: mean=2.3456, std=1.0234

Hooks capture intermediate activations without modifying the model code.

Conclusion

Hooks enable model introspection and surgery: extracting features, modifying gradients, and debugging. They're powerful but require careful management to avoid memory leaks. Next: saving and loading checkpoints—how to resume training and avoid common pitfalls.