Vision Transformers and LLMs excel separately. LLaVA connects them: use a frozen vision encoder (CLIP) + frozen LLM + trainable projection layer. This simple architecture achieves multimodal understanding without expensive retraining.
Architecture
Code Snippet
Image → Vision Encoder (CLIP ViT) → [d=768]
↓
Projection Layer (Linear)
↓
[d=4096]
↓
Combined with prompt text → LLaVA LLM → Caption/Answer
Implementation
Code Snippet
import torch
import torch.nn as nn
from transformers import CLIPVisionModel, AutoModelForCausalLM
class LLaVA(nn.Module):
def __init__(self, vision_model_name, llm_model_name):
super().__init__()
# Frozen vision encoder (CLIP)
self.vision_encoder = CLIPVisionModel.from_pretrained(vision_model_name)
self.vision_encoder.eval()
for param in self.vision_encoder.parameters():
param.requires_grad = False
# Frozen LLM
self.llm = AutoModelForCausalLM.from_pretrained(llm_model_name)
self.llm.eval()
for param in self.llm.parameters():
param.requires_grad = False
# Only trainable: projection layer
vision_dim = self.vision_encoder.config.hidden_size # 768
llm_dim = self.llm.config.hidden_size # 4096
self.projection = nn.Linear(vision_dim, llm_dim)
def forward(self, images, input_ids, attention_mask=None):
"""
images: (batch, 3, 224, 224)
input_ids: (batch, seq_len) tokenized text
"""
# Extract image features
with torch.no_grad():
image_features = self.vision_encoder(images).pooler_output # (batch, 768)
# Project to LLM space
projected_features = self.projection(image_features) # (batch, 4096)
# Embed text
with torch.no_grad():
text_embeddings = self.llm.get_input_embeddings()(input_ids) # (batch, seq_len, 4096)
# Combine: [image_feature, text1, text2, ...]
combined_embeddings = torch.cat([
projected_features.unsqueeze(1),
text_embeddings
], dim=1) # (batch, 1 + seq_len, 4096)
# LLM forward
with torch.no_grad():
outputs = self.llm(inputs_embeds=combined_embeddings, ...)
return outputs
Training Strategy
Code Snippet
Stage 1: Pre-training
- Align vision and language: use image-caption pairs
- Only train projection layer
- Training: a few days on single GPU
Stage 2: Fine-tuning (optional)
- Train on instruction-following data
- 1-2 days on single GPU
Benchmarks
Code Snippet
Model: LLaVA-13B vs GPT-4V vs Claude
Benchmark: MMVP, GQA, LLaVA-Bench
LLaVA-13B:
- MMVP: 61.3%
- GQA: 92.7%
- Instruction: 80.3%
GPT-4V (proprietary):
- MMVP: 88%
- GQA: 95%
- Instruction: 92%
LLaVA is 90% of GPT-4V quality at 1/100 the parameters!
Why This Works
- Vision encoders are pre-trained: CLIP already understands images
- LLMs are pre-trained: Already understand text
- Minimal bridge needed: Simple projection connects them
- Alignment is cheap: Only train 0.01% of parameters
Conclusion
LLaVA demonstrates that multimodal models don't require expensive retraining. Combining frozen components with a small trainable projection is elegant and effective. This approach powers many open-source multimodal systems. Next: exploring more advanced architectures.