PYTORCH

Profiling and Bottleneck Detection: torch.profiler, Memory Snapshots, and GPU Utilization Analysis

Training is slow somewhere. Learn torch.profiler to identify bottlenecks: GPU kernel time, CPU overhead, memory allocations, and data loading delays.

Soham Sharma

AI Engineer, Botmartz · October 19, 2026 · 12 min read

◈ Open in Google Colab ⌥ View Code on GitHub

Read Time

12 min

Failure Modes

Code Snippets

Runnable Notebook

Profiling and Bottleneck Detection: torch.profiler, Memory Snapshots, and GPU Utilization Analysis

Retrieval Quality Chunking Strategy Hybrid Search Continuous Evaluation Guardrails Takeaways

You've optimized learning rates, mixed precision, and distributed training, but training is still slow. Before blindly guessing, profile your code. Measure kernel time, CPU overhead, memory allocations, and I/O waits. This post covers torch.profiler: how to identify bottlenecks and which metrics matter.

Basic Profiling

Code Snippet

import torch
import torch.nn as nn
from torch.profiler import profile, record_function

model = nn.Linear(1000, 1000).cuda()
x = torch.randn(128, 1000, device='cuda')

with profile(activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA]) as prof:
    with record_function("model_inference"):
        y = model(x)

print(prof.key_averages().table(sort_by="cuda_time_total"))

Output:

Code Snippet

Name                CPU Time    CUDA Time   Count
───────────────────────────────────────────────
model_inference     100μs       50000μs     1
add                 20μs        100μs       1
linear              50μs        49800μs     1

The linear operation takes 49.8ms on GPU, which is where time is spent.