- RAG
- Data governance
Six steps to production-grade RAG: from document parsing to cited answers
By Apolla delivery team3 min read
Key takeaway
RAG answer quality is determined by six stages working together: document parsing, semantic chunking, embedding, hybrid retrieval, reranking with compression, and generation with citations. In practice, about 60% of "inaccurate answer" problems originate before retrieval (in parsing and chunking), not in the model. Only a 100–300 item evaluation set with reference answers, measuring recall, rerank hit rate and faithfulness layer by layer, can isolate the specific stage at fault.
Step one: document parsing
Enterprise documents are full of tables, multi-column layouts and scanned pages. A generic parser that scrambles a table into disordered text is where every downstream stage starts to fail. You need OCR, layout analysis and table reconstruction, a suitable parsing path for each of PDF, Word, PowerPoint and images, and preserved metadata such as heading hierarchy and page numbers.
Step two: semantic chunking
Splitting by a fixed character count cuts a complete policy clause in half. It is better to chunk by meaning and heading hierarchy, add overlapping windows, and use parent–child chunks: retrieve on small chunks for precision, then pass the parent chunk to the model for complete context.
Step three: embedding
General-purpose embedding models represent industry terminology and internal abbreviations poorly. Once you have some labeled data, fine-tuning the embedding model on domain data typically lifts recall by 5–15 percentage points. Multi-vector representation (a summary vector and a full-text vector for the same chunk) works better for long documents.
Step four: hybrid retrieval
Vector search is strong at semantic similarity; BM25 is strong at exact matches on model numbers, IDs and names. Fusing the two and adding query rewriting (synonym expansion, HyDE) is the stable approach today. Knowledge graphs fill the gap where multi-hop reasoning is needed ("who heads the parent department of department A?").
Step five: reranking and compression
After recalling 20–50 candidate passages, a rerank model orders them precisely and the top 5–8 go to generation. Context compression strips sentences unrelated to the question, which saves tokens and reduces the chance that irrelevant content pulls the model off course.
Step six: generation with citations
Require the model to tag each cited passage by number in its answer, and render those tags as clickable links to the source text. A faithfulness check verifies that every claim in the answer is supported by a retrieved passage; anything unsupported is marked "unverified" rather than output as fact.
- 60%
- of "inaccurate answer" problems originate before retrieval
- 5–15pts
- recall lift from domain fine-tuned embeddings
- 100–300items
- starting size of an evaluation set
Apolla delivery statistics
Locate the fault layer by layer with an evaluation set
Check recall first (did we find the passages we should have?), then reranking (are the relevant passages ranked at the top?), and finally generation (is the model faithful to what was retrieved?).
Only when each layer reports its own metric can you tell whether to fix the parser, swap the embedding model or adjust the prompt. RAG optimization without layered metrics is, in essence, guesswork.