Paper: "LoRA: Low-Rank Adaptation of Large Language Models" (Hu et al., 2021) ArXiv: https://arxiv.org/abs/2106.09685 Key Insight: Instead of fine-tuning all parameters, add small trainable matrices to frozen weights. Uses 99.9% fewer parameters, 10× less memory, and achieves competitive results.
The Problem: Fine-Tuning is Expensive
Full Fine-Tuning a 7B LLaMA:
- All 7B parameters are trainable
- Gradient memory: 7B × 4 bytes × 4 (FP32 forward + backward) = 112GB
- Can't fit on single GPU
LoRA Alternative:
- Freeze all parameters
- Add low-rank matrices (~0.1% parameters)
- Only 10-100MB GPU memory needed
LoRA Mechanism
For a weight matrix W, instead of learning ΔW directly, learn:
W' = W + ΔW
ΔW = A × B
where:
W is (d_out, d_in) frozen
A is (d_out, r) trainable
B is (r, d_in) trainable
r << min(d_out, d_in) (low rank)
The key: multiplying small matrices is much cheaper than updating large weights.
Implementation
import torch
import torch.nn as nn
class LoraLayer(nn.Module):
def __init__(self, in_features, out_features, rank=8, alpha=16):
super().__init__()
self.rank = rank
self.alpha = alpha
# Original weight (frozen)
self.weight = nn.Parameter(
torch.empty(out_features, in_features),
requires_grad=False
)
# LoRA matrices (trainable)
self.lora_A = nn.Parameter(torch.randn(in_features, rank) / rank)
self.lora_B = nn.Parameter(torch.zeros(rank, out_features))
def forward(self, x):
# x: (batch, ..., in_features)
# Original weight + low-rank update
# y = x @ W.T + (x @ A) @ B.T * (alpha / rank)
output = x @ self.weight.t()
output += (x @ self.lora_A) @ self.lora_B.t() * (self.alpha / self.rank)
return output
class LoraLinear(nn.Module):
"""Replace nn.Linear with LoRA version"""
def __init__(self, in_features, out_features, rank=8):
super().__init__()
self.linear = nn.Linear(in_features, out_features)
self.lora_A = nn.Parameter(torch.randn(in_features, rank) / rank)
self.lora_B = nn.Parameter(torch.zeros(rank, out_features))
self.rank = rank
def forward(self, x):
return self.linear(x) + (x @ self.lora_A) @ self.lora_B.t()
# Usage
base_model = AutoModelForCausalLM.from_pretrained('llama-7b')
# Freeze all parameters
for param in base_model.parameters():
param.requires_grad = False
# Replace some Linear layers with LoRA
for name, module in base_model.named_modules():
if isinstance(module, nn.Linear) and any(x in name for x in ['attention', 'mlp']):
# Convert to LoRA
lora_module = LoraLinear(
module.in_features,
module.out_features,
rank=8
)
# Copy weight
lora_module.linear.weight.data = module.weight.data.clone()
# Replace in model
parent = base_model
attr_name = name.split('.')[-1]
setattr(parent, attr_name, lora_module)
# Now fine-tune with tiny memory footprint
optimizer = torch.optim.Adam(
[p for p in base_model.parameters() if p.requires_grad],
lr=1e-4
)
for epoch in range(3):
for batch in finetune_dataset:
loss = base_model(batch)
loss.backward()
optimizer.step()
optimizer.zero_grad()
Memory Comparison
Fine-tuning LLaMA 7B:
Full Fine-Tuning:
- Trainable params: 7,000M
- Gradient memory: 112GB
- Inference: Full 7B loaded
LoRA (r=8):
- Trainable params: 7.1M (0.1% of total!)
- Gradient memory: 56MB
- Inference: Same as base (can merge LoRA into W)
Benchmark Results
Task: Text generation (WikiQA, WikiBio)
GPT-3 (175B): 100% (baseline)
Full FT on 7B: 87.5%
LoRA on 7B (r=8): 87.1% (only 0.4% worse!)
LoRA on 7B (r=16): 87.4% (competitive!)
LoRA matches full fine-tuning with 99.9% fewer parameters!
Why Low-Rank Works
Hypothesis: Fine-tuning also results in low-rank updates.
Intuition:
- Language models already have learned rich representations
- Fine-tuning for a new task is a small adaptation
- This adaptation likely lives in a low-rank subspace
- So we don't need to update all parameters, just the directions that matter
This is why LoRA works so well—fine-tuning naturally has low rank!
Our Analysis
LoRA is revolutionary because it makes LLM fine-tuning democratized. Before LoRA, you needed massive GPUs to fine-tune. Now, anyone can fine-tune a 7B model on a single GPU (24GB VRAM). This is the reason open-source LLM fine-tuning exploded.
Practical Tip: LoRA is now standard:
# HuggingFace integration
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=8,
lora_alpha=16,
target_modules=['q_proj', 'v_proj'],
lora_dropout=0.05
)
model = get_peft_model(base_model, config)
References
- Paper: LoRA (Hu et al., 2021)
- Extensions: QLoRA (4-bit + LoRA), DoRA (directional), etc.
- Implementation: https://github.com/microsoft/LoRA
Conclusion
LoRA proves that you don't need to update all parameters to adapt large models. This opened the door for widespread LLM customization. Understanding the low-rank hypothesis is key to understanding why this works and how to improve it. Next: we'll explore multimodal research and vision-language models.