Short answer
Retrieval-augmented generation makes an assistant answer from your own documents by searching them at question time and giving the matching passages to the model. The technical build is fast. The work that decides whether it succeeds is choosing which documents are authoritative, removing the ones that contradict them, enforcing who is allowed to see what, and testing answers against a set of questions whose correct answers you already know.
The mechanism, in one paragraph
Your documents are split into passages and indexed so they can be searched by meaning rather than by exact wording. When a question arrives, the system searches that index, takes the best few passages, and sends them to the model together with the question and an instruction to answer only from what it was given. The model composes the answer; the facts come from your files.
Two consequences follow, and both are useful. Updating a document updates the answers immediately, with no retraining. And because you know which passages were used, you can show the source — which is what turns an answer someone has to trust into one they can verify.
The approach is not new and it is not a vendor invention. It was named and described in the 2020 paper that introduced retrieval-augmented generation, and the architecture in production today is recognizably the one set out there. That matters practically: this is a well-understood technique with known failure modes, not a frontier anyone needs to be first to.
A useful reframing: retrieval is a search problem with a language model attached to the end. Almost every disappointing result traces back to the search half, not the model half.
The corpus is the project
Here is what nobody scopes for. Most companies do not have a set of documents that agree with each other. They have a shared drive containing the current procedure, the previous procedure, a draft someone wrote during a reorganization, and a version a client was sent two years ago with special terms.
A human navigating that folder applies context: they know which file is current because they know who wrote it and when. A retrieval system has no such context. It returns whichever passage best matches the question, and the outdated procedure often matches better because it was written more directly.
So the first phase of any serious project is editorial, not technical. Decide which documents are authoritative. Remove or clearly mark the rest. Give each one an owner and a review date. This is tedious, it is the actual work, and every hour spent on it is worth several spent tuning retrieval parameters afterwards.
Why retrieval fails before the model does
When an answer is wrong, the instinct is to blame the model. In our experience the failure is upstream in most cases, and it comes in a few recognizable shapes.
- Passages split badly. A table cut in half, or a clause separated from the condition that governs it. The retrieved text is technically relevant and materially misleading.
- Vocabulary mismatch. Staff ask about "holiday" and the policy says "annual leave". Semantic search handles much of this, but not domain jargon it has never seen.
- The answer is spread across documents. Half in the contract, half in an addendum. Retrieval returns one and the answer is confidently incomplete.
- Scanned documents. A PDF that is an image of text contains no text. It indexes as nothing and silently contributes nothing.
The diagnostic is straightforward and skipped surprisingly often: when an answer is wrong, look at which passages were retrieved. If the correct passage was not among them, no change to the prompt or the model will help.
Permissions: the requirement discovered too late
A pilot built on a folder of general documentation works, everyone is pleased, and someone asks the obvious question: can it also answer from HR files, or the client folders, or the board pack?
At that point the system needs to know who is asking and what they are allowed to see, enforced at retrieval time rather than as a filter on the answer. Retrofitting that is significantly harder than building it in, because it touches how documents are indexed, not just how they are served.
Decide early whether the assistant will ever cover restricted material. If the answer is even possibly yes, design for identity from the first version — even if version one only serves public documents.
How to know whether it is right
Impressions do not survive contact with production. Build a test set: fifty questions people genuinely ask, with the correct answer and its source document written down by someone who knows. Run it after every meaningful change.
Score two things separately, because they fail independently. Did retrieval return the right passage? Did the answer faithfully reflect it? Mixing them together makes it impossible to tell whether you have a search problem or a generation problem, and they have different fixes.
Track refusals as a positive signal, not a defect. An assistant that says "that is not in the documents I have" is behaving correctly, and a system with zero refusals is almost certainly inventing things.
Where it is worth doing at all
The honest test is volume and repetition. If the same questions are asked weekly by different people, and the answers exist in writing but take a while to find, this pays for itself. Internal procedures, product documentation, contract terms and technical specifications all fit that shape.
It is a poor fit where the answer requires judgment rather than lookup, where the documents are few enough that a person could reasonably read them, or where being wrong once has consequences nobody will accept. In those cases the honest recommendation is better search, or better documentation, and no model at all.
If your use case is customer-facing rather than internal, the constraints tighten considerably — the escalation design matters more than the retrieval quality, which we cover in AI chatbots for customer service. For the orchestration around it — receiving the document, calling the model, writing the result back to a real system — see the practical guide to n8n. And if you are weighing this against a fully agentic design, see what AI agents actually do.
Frequently asked questions
What does RAG actually stand for, in practice?
Retrieval-augmented generation. The mechanism is simpler than the name: when someone asks a question, the system searches your documents for relevant passages, puts those passages in front of the model along with the question, and asks it to answer from them. The model is not trained on your data and does not memorize it. It reads the passages at the moment of the question, the same way a person would look something up before answering.
Is fine-tuning a model on our documents better?
For factual question answering, almost never. Fine-tuning teaches a model a style or a task, not a set of facts you can update. Your documents change; a fine-tuned model does not, short of retraining. Retrieval also lets you show which document an answer came from, which is what makes the output checkable. The rule of thumb: fine-tune to change how the model behaves, retrieve to change what it knows.
Why does it give wrong answers even with our documents loaded?
Usually because retrieval failed before the model was involved. If the passage containing the answer was never returned, the model answers from nothing and produces something plausible. The second common cause is a corpus containing several versions of the truth — an outdated procedure sitting alongside the current one, both equally retrievable. The model has no way to know which is authoritative unless you tell it.
How long does it take to build something usable?
A working prototype over a small, clean set of documents is genuinely quick — days rather than months. What takes time is everything around it: deciding which documents are authoritative, removing the ones that contradict each other, handling access rights so people only see what they are entitled to, and measuring whether the answers are actually right. Teams that budget for the prototype and not the corpus work are the ones whose pilot never ships.
We build these, including the unglamorous corpus work that decides whether they are any good. Book a call and bring the questions your team asks most often.
