Fine-tuning a 7B parameter model requires storing gradients for all parameters, consuming enormous GPU memory. LoRA reduces this by updating only low-rank matrices, reducing memory by 10×. Understanding parameter-efficient fine-tuning is essential for practical LLM customization.
LoRA (Low-Rank Adaptation)
Instead of updating weight matrix W, update W + A·B where A and B are small (low-rank).
from peft import LoraConfig, get_peft_model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
config = LoraConfig( r=8, # Low-rank dimension lora_alpha=16, # Scaling factor target_modules=["q_proj", "v_proj"], # Which layers to update lora_dropout=0.05, )
model = get_peft_model(model, config)
# Fine-tune with 10× less memory
LoRA achieves 99% of full fine-tuning quality with 10× less memory.
QLoRA
Combine LoRA with quantization: 4-bit quantized model + LoRA updates.
from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig
# Quantize to 4-bit bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, )
model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b", quantization_config=bnb_config )
# Add LoRA config = LoraConfig(r=8, target_modules=["q_proj", "v_proj"]) model = get_peft_model(model, config)
QLoRA fits a 13B model on a single GPU (8GB VRAM).
Conclusion
Parameter-efficient fine-tuning (LoRA, QLoRA) makes customizing LLMs practical and cost-effective. LoRA is the standard for most use cases; QLoRA for extreme memory constraints. Understanding these techniques enables building domain-specific models affordably. Next: evaluating and comparing LLMs.