LLM evaluation is nuanced. Perplexity measures prediction accuracy on test data. BLEU and ROUGE measure generation quality (e.g., machine translation, summarization). Task-specific metrics (exact match, F1) measure downstream performance. Human evaluation remains the gold standard.
Perplexity
Lower perplexity = better language model. Measures how well the model predicts the test set.
# Perplexity = exp(mean(loss))
loss = -log P(text)
perplexity = math.exp(loss)
Perplexity measures intrinsic quality but doesn't measure usefulness.
BLEU Score
Measures overlap between generated and reference text.
from nltk.translate.bleu_score import sentence_bleu
reference = ["the", "cat", "sat", "on", "the", "mat"]
hypothesis = ["the", "cat", "sat", "on", "a", "mat"]
score = sentence_bleu([reference], hypothesis) # 0.75
BLEU is useful for machine translation and paraphrasing.
Task-Specific Metrics
For QA tasks, use exact match (EM) and F1 score.
# EM: Is the answer exactly correct?
em = 1 if predicted == reference else 0
# F1: Overlap between predicted and reference
precision = overlap / len(predicted)
recall = overlap / len(reference)
f1 = 2 * precision * recall / (precision + recall)
Conclusion
Metrics guide model development. Perplexity measures language modeling quality. Task-specific metrics measure downstream performance. Combining automated metrics with human evaluation ensures reliable assessment. Next: we'll explore multi-modal LLMs—extending LLMs to understand images and other modalities.