TL;DR
RAG quality depends on retrieval quality. If your chunks are too big, too small, lack context, or ignore document structure, even the best LLM will hallucinate. Fix chunking first—it's 80% of RAG success.
Retrieval-Augmented Generation (RAG) has become the go-to pattern for enterprise AI applications. Feed documents into a vector database, retrieve relevant chunks, pass them to an LLM. Simple in theory.
In practice, we see teams struggle with poor retrieval quality. The LLM is fine. The vector database is fine. The problem is almost always how documents were chunked before embedding.
Here are the four chunking mistakes we see most often—and how to fix them.
Mistake #1: Fixed-Size Chunks That Ignore Meaning
The easiest approach: split every document into 500-token chunks. Fast to implement, terrible for retrieval.
Fixed-size chunking cuts sentences in half, separates questions from answers, and splits tables across multiple chunks. The embedding captures fragments, not concepts.
[NEXT CHUNK]
"policyholder must submit Form 27-B with supporting documentation within 30 days..."
Fix: Use semantic chunking that respects paragraph and section boundaries. LangChain's RecursiveCharacterTextSplitter with custom separators is a good start.
Mistake #2: Chunks Without Context
A chunk says "The rate is 3.5% for the first year." Great—but 3.5% of what? Which product? Which customer segment? The chunk is meaningless without its parent context.
When users ask "What's the interest rate for commercial loans?", this chunk might match on "rate"—but the LLM has no way to verify it's actually about commercial loans.
Fix: Prepend context to each chunk. Include the document title, section header, and any relevant metadata:
The rate is 3.5% for the first year, increasing to prime + 1.5% thereafter."
Mistake #3: Tables Treated as Text
Standard text splitters mangle tables. Columns become jumbled text. Relationships between cells disappear. A pricing table with 20 rows becomes an unreadable mess that still somehow matches user queries.
Fix: Detect and extract tables separately. Convert them to structured formats (JSON or Markdown tables) before chunking. For complex tables, consider storing them as separate retrievable objects with a natural language summary.
Pro tip: For PDFs with complex tables, use a specialized extraction tool like Camelot or Tabula before chunking. A document-aware extraction workflow can handle this before retrieval indexing.
Mistake #4: No Chunk Overlap
With zero overlap, information at chunk boundaries becomes unretrievable. A user asks a question that spans two chunks—neither chunk alone is relevant enough to surface.
Fix: Use 10-20% overlap between chunks. If your chunks are 500 tokens, overlap by 50-100 tokens. This creates redundancy but dramatically improves retrieval for boundary cases.
Balance is key: too much overlap bloats your vector database and increases noise. Too little creates retrieval gaps.
Quick Diagnostic
Before blaming the LLM or switching embedding models, test your chunking:
- Pick 10 real user questions that your RAG system answered poorly
- Manually search your chunks—can you find the right answer?
- If the answer exists but in a mangled or context-free chunk, fix chunking
- If the answer isn't in any chunk, fix your document ingestion
We find chunking issues in about 80% of RAG debugging sessions. The remaining 20% split between embedding model selection and retrieval parameters (top-k, similarity thresholds).
