Retrieval-augmented generation — RAG — is how most useful AI features get built: fetch the relevant context from your own data, hand it to a model, and let it answer grounded in real information rather than its training. The first version takes an afternoon. Making it good enough that people rely on it takes considerably longer, and the effort goes somewhere that surprises teams the first time.
The model is rarely the bottleneck. The retrieval is.
Garbage retrieved, garbage generated
A model can only reason over what you give it. If retrieval returns the wrong three paragraphs, the answer will be confidently wrong, and no amount of prompt tuning fixes that. Most "the AI is hallucinating" complaints are really "the retrieval fetched the wrong context" complaints.
Chunking decides what can be found
You have to split documents into pieces to retrieve them, and how you split matters more than it looks. Chunks that are too big bury the relevant sentence in noise; too small and they lose the context that made them meaningful. There is no universal setting — it depends on your documents, and it is worth testing rather than guessing.
Similarity is not the same as relevance
Vector search finds text that is semantically close to the question. Close is not always relevant. The strongest systems combine semantic search with keyword matching, and often re-rank the results before passing them on, so an exact term or a recent record is not lost just because its wording differed.
- Retrieve more than you need, then re-rank down to the best few.
- Combine keyword and semantic search — each catches what the other misses.
- Respect metadata: recency, permissions, and source often matter as much as similarity.
You cannot improve what you do not measure
The thing that turns a RAG prototype into a dependable feature is evaluation. Build a set of real questions with known good answers, and measure whether the system retrieves the right context and produces the right response as you change it. Without this, every "improvement" is a guess, and you will not notice when a change quietly makes things worse.
Spend your time on retrieval quality and evaluation. That is where a RAG feature goes from demo to something people trust.
Keep the source of truth outside the model
The model reads from your data; it should never become the record. Keep the authoritative version in your own system, retrieve from it at request time, and show users where an answer came from. That is what lets someone trust the output — or check it when it matters.