Most conversations about RAG end at the demo. The agent answers the question, the citation appears underneath it, everyone nods. The bit that never makes the slide is the moment the retrieval step returned three documents instead of the right one, and the model picked its favourite anyway.
That moment is the design. Everything else is plumbing.
#What RAG actually is
RAG isn't the model getting smarter. It's a search step bolted on in front of it. Given a question, the system searches your documents for the passages most likely to be relevant, hands those passages to the model, and asks it to answer from them instead of from memory.
The model still has to do two things well: recognise which of the retrieved passages actually answer the question, and say so honestly when none of them do. Retrieval finds candidates. It doesn't guarantee one of them is right.
That's the whole point of RAG. It's also the thing that makes it risky if you don't put edges around it.
#The retrieval step is the product
We build RAG agents that search a business's actual documents, the handbook, the spec, the price list, before they answer anything. The easy part is generating a fluent response. The part that actually matters is what happens when the search comes back thin: two loosely related passages, nothing on point, or a document that's since been superseded.
A properly built agent doesn't quietly do its best with whatever it found. It checks whether the retrieved passages actually support an answer, and says when they don't.
If your RAG agent always finds something to cite, you don't have grounded answers. You have search results wearing a confident sentence.
#Three questions before you trust an answer
We won't let a RAG agent answer a live question until we can answer these three, in plain English, to the person relying on it:
- What's actually in the index, and how current is it? If nobody can say when the documents were last refreshed, the citation is decoration, not proof.
- What does it do when retrieval comes back weak? Say so, or guess anyway, that decision needs to be written down before the agent goes live, not discovered the first time it happens.
- Who checks the citation, and how easily can they? A source that takes five minutes to verify might as well not be there. The citation needs to point at a specific passage, not a five-hundred-page PDF.
#When RAG is the wrong answer
There are questions RAG adds nothing to. If the answer is genuinely general knowledge, unrelated to anything specific to your business, retrieval just adds latency for no benefit. If your documents are so out of date or so poorly structured that nothing in the index is trustworthy, RAG will confidently cite the wrong thing rather than fixing the underlying mess.
RAG earns its place when the answers exist in writing somewhere in your business, the writing changes often enough that a model can't just memorise it, and someone needs that answer faster than they can search for it themselves.
#Where to start
Pick one document your team checks constantly, a handbook, a spec, a policy file. Write down the last time someone got a confidently wrong answer because they were guessing instead of checking it. That's usually where RAG fits, and that's usually the document that was never going to stay memorised anyway.
If you can't point to a specific document, RAG isn't the first thing to build. The documentation is.
Recognise any of this in your own business? Tell us about it and we’ll say whether it’s worth automating.
Talk through your process
