Training SNNs with standard backpropagation fails because spikes are discrete (0/1), making gradients zero everywhere. Surrogate gradients solve this: pretend the spike function is smooth during backprop, but use discrete spikes during forward pass. This trick enables training SNNs like standard neural networks.
The Surrogate Gradient Method
import torch
import torch.nn as nn
class SurrogateSpike(torch.autograd.Function): @staticmethod def forward(ctx, x): # Forward: discrete spike spike = (x > 0).float() ctx.save_for_backward(x) return spike
@staticmethod def backward(ctx, grad_output): # Backward: smooth surrogate (sigmoid-like gradient) x, = ctx.saved_tensors # Surrogate: fast sigmoid (smooth approximation) grad = 1.0 / (1.0 + torch.abs(x)) ** 2 # Fast sigmoid surrogate return grad_output * grad
class SurrogateLIF(nn.Module): def __init__(self, leak=0.99, threshold=1.0): super().__init__() self.leak = leak self.threshold = threshold self.V = None
def forward(self, x): if self.V is None: self.V = torch.zeros_like(x)
# Integrate self.V = self.leak * self.V + x
# Spike with surrogate gradient spike = SurrogateSpike.apply(self.V - self.threshold)
# Reset self.V = self.V * (1 - spike)
return spike
Training Example
import torch.optim as optim
device = torch.device('cuda') snn = SNNNetwork().to(device) optimizer = optim.Adam(snn.parameters(), lr=1e-3) criterion = nn.CrossEntropyLoss()
for epoch in range(100): for images, labels in train_loader: images, labels = images.to(device), labels.to(device)
# Convert to spike trains (Poisson encoding) spike_train = (torch.rand_like(images).expand(32, 10, -1) < images.unsqueeze(1)).float()
# Forward output = snn(spike_train) # (batch, 10 timesteps, 10 classes)
# Decode: average spike count per class pred = output.mean(dim=1) # (batch, 10) loss = criterion(pred, labels)
# Backward (with surrogate gradients) optimizer.zero_grad() loss.backward() optimizer.step()
snn.reset() # Reset membrane potentials
Surrogate Functions
Different surrogates have different properties:
- Fast Sigmoid: 1 / (1 + |x|)^2
- Fast, smooth, good for STDP-like rules
- Error Function: erf(x / sqrt(2))
- Mathematically principled
- Exponential: exp(-|x|)
- Sharper, more localized
Choose based on empirical performance on your task.
Conclusion
Surrogate gradients enable training SNNs with standard backpropagation. The forward/backward asymmetry is the key: discrete spikes forward, smooth surrogate backward. This pragmatic solution makes SNNs trainable and competitive with ANNs. Next: Neuromorphic hardware and deployment.