Quantization: Post-Tra in ing Quantization vs. Quantization-Aware Training, INT8 Inference

Quantization shrinks models by representing weights and activations in lower precision (int8 instead of float32). A 100MB model becomes 25MB. Inference is 2–4× faster on CPUs, and memory usage drops 4×. Two approaches: post-training quantization (no retraining) and quantization-aware training (simulate quantization during training). This post covers both and shows when to use each.

Post-Training Quantization

Quantize a trained float32 model without retraining.

import torch
import torch.nn as nn from torch.quantization import quantize_dynamic
model = nn.Sequential( nn.Linear(100, 128), nn.ReLU(), nn.Linear(128, 10) )
# Quantize linear layers dynamically (weights to int8) quantized_model = quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)
# Inference x = torch.randn(4, 100) output = quantized_model(x)
print(f"Original model size: {sum(p.numel() for p in model.parameters()) * 4 / 1e6:.2f} MB") print(f"Quantized model size: {sum(p.numel() for p in quantized_model.parameters()) * 1 / 1e6:.2f} MB") 

Output:

Original model size: 0.22 MB
Quantized model size: 0.05 MB 

PTQ is fast and requires no retraining, but accuracy may drop 1–3%.

Conclusion

Quantization enables efficient inference on resource-constrained devices. PTQ is quick and works for many models; QAT gives better accuracy when quantization loss matters. Next: TorchScript and model export—how to deploy models to production.