Accuracy metrics are computed offline; production metrics matter online. Track latency, throughput, error rates, and data drift. This post covers monitoring infrastructure, drift detection, and alerting rules.
Latency Tracking
Code Snippet
import time
from collections import deque
class LatencyMonitor:
def __init__(self, window_size=1000):
self.latencies = deque(maxlen=window_size)
def record(self, latency_ms):
self.latencies.append(latency_ms)
def p99_latency(self):
sorted_lats = sorted(self.latencies)
return sorted_lats[int(0.99 * len(sorted_lats))]
monitor = LatencyMonitor()
# Record latencies during inference
monitor.record(25)
monitor.record(30)
print(f"P99 latency: {monitor.p99_latency()}ms")
Conclusion
Production monitoring detects issues early. Latency, throughput, drift, and error rates provide comprehensive visibility. Building monitoring infrastructure separates hobbyist ML from production-grade systems. This concludes the PyTorch Mastery series.