LLM inference is I/O bound: each token generation requires matrix multiplications that are small relative to memory transfer. Optimize with quantization (reduce model size), batching (amortize I/O), KV cache (avoid recomputation), and speculative decoding (predict multiple tokens ahead).
Quantization
Reduce model precision: float32 → float16 → int8. Smaller models fit in smaller memory, reducing latency.
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
# Quantize to int8 quantized_model = torch.quantization.quantize_dynamic( model, {torch.nn.Linear}, dtype=torch.qint8 )
# 4× smaller, faster inference
KV Cache
During generation, attention recomputes all keys/values. Cache them instead.
Without KV cache: Each token recomputes attention for all previous tokens. O(n²)
With KV cache: Store keys/values, reuse. O(n)
This reduces latency 4-8× for long sequences.
Speculative Decoding
Generate multiple tokens ahead speculatively, then verify.
1. Guess next 5 tokens using a fast model
- Verify with main model: which guess is correct?
- Keep correct tokens, speculate on remainder
This reduces latency by 2-3× with minimal accuracy loss.
Conclusion
Inference optimization is essential for production LLM serving. Each technique (quantization, KV cache, speculative decoding, batching) addresses a specific bottleneck. Combined, they enable real-time, cost-effective inference. Next: deploying LLMs—how to serve models reliably at scale.