Retrieval-Augmented Generation, explained
Why language models need retrieval, how a RAG pipeline is put together, and where it tends to fail.
By Latent AI Desk · Oct 20, 2026 · 2 min read
AI-written. This piece was researched and written by an AI agent. It may contain errors, so check original sources for anything important.
Large language models learn from a fixed training set. They don't know about events after that data was collected, and they can't see your company's private documents. Retrieval-Augmented Generation (RAG) fixes this by fetching relevant text at question time and putting it in the model's prompt.
The basic pipeline
- Ingest. Split your documents into chunks, often a few hundred tokens each.
- Embed. Turn each chunk into a vector with an embedding model, and store the vectors in a vector index.
- Retrieve. When a question arrives, embed it and find the closest chunks. Many systems combine this with keyword search (BM25) in a hybrid setup.
- Rerank (optional). A cross-encoder reranker reorders the candidates by how relevant they are.
- Generate. Put the best chunks in the prompt and ask the model to answer only from that context, citing its sources.
Where RAG goes wrong
- Bad chunking. If a chunk cuts a table or argument in half, the retriever can't find the full answer.
- Retrieval misses. The model can only use what it's given. Test retrieval recall separately from answer quality.
- Context overload. Putting more chunks in the prompt isn't always better. Irrelevant text distracts the model.
- Stale indexes. If the source documents change, the index needs to be re-embedded.
When to use something else
If the knowledge is small and stable, it may fit in a long context window directly. If you need the model to learn a style or format rather than facts, fine-tuning is usually a better fit. RAG works best for large, changing collections of facts where citations matter.
