What you'll learn
Quick Answer
RAG (retrieval-augmented generation) is a pattern for answering questions from your own documents. Ahead of time you split the documents into chunks, convert each chunk to an embedding vector, and store the vectors. At query time you embed the user's question, find the chunks whose vectors are closest, and paste those chunks into the prompt as context. The model then answers from that context instead of from memory. Nothing is retrained - RAG is retrieval plus a bigger prompt.
The problem RAG solves
A model's knowledge is frozen at its training cutoff and never included your private data. Ask a bare model "what is our refund window for enterprise customers" and it produces a confident, generic, wrong answer - it has no way to know, and it is not built to say so.
You could fine-tune, but that is slow, costly, needs redoing every time a document changes, and still gives you no citations. RAG takes the other route: leave the model alone and improve the prompt. If the relevant paragraph from your policy doc is sitting in the context, the model can just read it.
Before RAG: "Our standard refund window is 30 days" - invented. After RAG, with the policy chunk retrieved: "Enterprise refunds follow a 60-day window per section 4.2" - lifted from your actual document, with a link back to it so the user can check.
Indexing: chunk, embed, store
This runs once, and again whenever documents change.
- Chunk. Split each document into pieces small enough to be about one thing - often a few hundred words, sometimes split on paragraphs or headings. Too large and a chunk covers several topics, so retrieval is imprecise and you waste tokens; too small and a chunk loses the context that made it meaningful. A common tactic is a small overlap between adjacent chunks so a sentence split across a boundary is not lost.
- Embed. Run each chunk through an embedding model, which returns a fixed-length vector of numbers that encodes meaning. Chunks about similar things get similar vectors.
- Store. Put the vectors, with the original text and metadata (source, section, date, access level), into something that can do fast nearest-neighbour search. A dedicated vector database is common, but Postgres with a vector extension is fine for a first version.
Retrieval: find the closest chunks
When a question comes in, embed it with the same model you used for the chunks - this is not optional, vectors from different models are not comparable. Then ask the store for the top k chunks whose vectors are nearest the question's vector, usually ranked by cosine similarity. k is typically 3 to 10. Those chunks are your evidence.
Many production systems add a second pass: over-fetch, say, 30 candidates cheaply, then re-rank them with a slower, more accurate model and keep the best 5. You can also filter by metadata first - only this customer's documents, only pages updated this year, only content the user is allowed to see - before the vector search runs. That last one matters: without an access filter, RAG will happily quote a document the user should never have seen.
Generation: answer from the context
Assemble a prompt that contains the retrieved chunks and the question, with an instruction tying them together: "Answer using only the context below. If the context does not contain the answer, say you do not know. Cite the section you used."
The model reads the chunks and responds. Because the evidence is right there, answers are grounded and you can show the user which source each claim came from. This citation step is half the value of RAG in practice: users trust an answer they can verify, and when an answer is wrong you can look at exactly which chunk was retrieved and decide whether the bug is in retrieval or in the model's reading of good context.
Keep the retrieved context clearly separated from the question in the prompt, and cap how many chunks you include - more is not better once the useful one is in there, and every extra chunk adds cost and dilutes attention.
Vector search alone misses exact matches
Embedding similarity is great at "these mean the same thing" and bad at "these are literally the same string". Ask for error code E-4021, a specific SKU, a person's surname, or a function name, and pure vector search can rank a vaguely related paragraph above the page that contains the exact token, because the rare string barely moves the embedding.
The standard fix is hybrid search: run a keyword search (BM25 or plain full-text) and a vector search, then combine the two ranked lists. Keyword search nails exact identifiers and rare terms; vector search handles paraphrase and synonyms. Most managed vector stores now offer hybrid retrieval as a built-in option, and turning it on is often the single biggest quality jump after basic RAG is working.
RAG moves hallucination, it does not remove it
The weakest link is retrieval. If the nearest chunks are the wrong ones - bad chunking, a question phrased differently from the document, an embedding model that does not know your domain - the model gets misleading context and answers from it just as confidently as it would from memory. The failure is quieter than a bare hallucination because the answer looks sourced.
What helps: test retrieval on its own (for a set of real questions, is the right chunk in the top k?), keep chunks focused, keep the "say you do not know" instruction, and log every retrieval so you can see what the model was actually handed. Watch the "lost in the middle" effect too - models attend less to content buried in the centre of a long context, so put the strongest chunk first or last rather than dumping ten chunks in arbitrary order.