Document loaders ingest PDFs, web pages, and databases. Text splitters chunk documents into manageable pieces—balancing context length with semantic coherence. This post covers loaders, splitters, and chunking strategies for RAG systems.
Document Loaders
from langchain.document_loaders import PDFLoader, WebBaseLoader
# Load PDF pdf_loader = PDFLoader("document.pdf") documents = pdf_loader.load() print(f"Loaded {len(documents)} pages")
# Load webpage web_loader = WebBaseLoader("https://example.com") web_docs = web_loader.load() print(f"Loaded {len(web_docs)} documents from web")
Output:
Loaded 10 pages
Loaded 1 documents from web
Text Splitters
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=200, separators=["\n\n", "\n", " ", ""] )
chunks = splitter.split_text(long_text) print(f"Split into {len(chunks)} chunks")
Output:
Split into 45 chunks
Recursive splitting respects paragraph/sentence boundaries before character limits, preserving semantics.
Conclusion
Document loading and chunking are foundational for RAG. Recursive splitting with overlap preserves context while managing token limits. Understanding loader capabilities and chunking strategies enables building effective retrieval systems. Next: embeddings and vector stores—how to store and retrieve chunks.