Profil in g and Bottleneck Detection: torch.profiler, Memory Snapshots, and GPU Utilization Analysis

You've optimized learning rates, mixed precision, and distributed training, but training is still slow. Before blindly guessing, profile your code. Measure kernel time, CPU overhead, memory allocations, and I/O waits. This post covers torch.profiler: how to identify bottlenecks and which metrics matter.

Basic Profiling

import torch
import torch.nn as nn from torch.profiler import profile, record_function
model = nn.Linear(1000, 1000).cuda() x = torch.randn(128, 1000, device='cuda')
with profile(activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA]) as prof: with record_function("model_inference"): y = model(x)
print(prof.key_averages().table(sort_by="cuda_time_total")) 

Output:

Name                CPU Time    CUDA Time   Count
─────────────────────────────────────────────── model_inference     100μs       50000μs     1 add                 20μs        100μs       1 linear              50μs        49800μs     1 

The linear operation takes 49.8ms on GPU, which is where time is spent.

Conclusion

Profiling reveals where time is actually spent. Next: custom loss functions and how to implement numerically stable losses.