Quantization shrinks models by representing weights and activations in lower precision (int8 instead of float32). A 100MB model becomes 25MB. Inference is 2–4× faster on CPUs, and memory usage drops 4×. Two approaches: post-training quantization (no retraining) and quantization-aware training (simulate quantization during training). This post covers both and shows when to use each.
Post-Training Quantization
Quantize a trained float32 model without retraining.
import torch
import torch.nn as nn
from torch.quantization import quantize_dynamic
model = nn.Sequential(
nn.Linear(100, 128),
nn.ReLU(),
nn.Linear(128, 10)
)
# Quantize linear layers dynamically (weights to int8)
quantized_model = quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)
# Inference
x = torch.randn(4, 100)
output = quantized_model(x)
print(f"Original model size: {sum(p.numel() for p in model.parameters()) * 4 / 1e6:.2f} MB")
print(f"Quantized model size: {sum(p.numel() for p in quantized_model.parameters()) * 1 / 1e6:.2f} MB")
Output:
Original model size: 0.22 MB
Quantized model size: 0.05 MB
PTQ is fast and requires no retraining, but accuracy may drop 1–3%.
Conclusion
Quantization enables efficient inference on resource-constrained devices. PTQ is quick and works for many models; QAT gives better accuracy when quantization loss matters. Next: TorchScript and model export—how to deploy models to production.