The trick is not that the AI knows more. The trick is that it has to read less at once: it picks out the few passages from your material that match the question, and works only with those. What that means in practice is what this article is about – using one perfectly ordinary question as an example.
The problem
You ask an AI assistant: “What notice period did we agree with Müller Ltd?” The assistant does not know your contract. It knows general knowledge from its training – and in the worst case it will produce a plausible-sounding, wrong answer.
The obvious solution: you attach the contract to your question. That works beautifully as long as there are only a handful of documents. With five hundred contracts it becomes impractical, because you do not know in advance which one holds the answer. This is exactly where RAG comes in.
What RAG means
RAG stands for retrieval-augmented generation – roughly: “generating answers, enriched by looking things up on purpose”. At its core there are two steps:
- Looking up (with every question): the system picks out the matching passages – typically five to twenty sections.
- Answering: only those sections go to the AI together with your question. It formulates the answer and can say which document it came from.
And what about the preparation?
Many explanations put a third step in front of all this: split the documents up, calculate “fingerprints”, load them into a database. That is not part of RAG – it depends on how you have the system search:
| How the search works | Preparation needed? |
|---|---|
| Meaning-based search (semantic) | Yes. The content fingerprints have to be calculated and stored in advance. |
| Keyword search (full text) | Usually not. A search index often already exists and maintains itself; sometimes the files are searched directly. |
| Database query | No. The data is already structured. |
That is why it matters what RAG is not: it is not a particular piece of software, not a product you buy, and not necessarily a database you have to fill first.
RAG is the principle of “look it up first, then answer”. What you look things up with is a free decision – more on that in part 2.
The real gain: the AI has to read less
This is the point that often gets lost. A worked example: let us assume your body of knowledge amounts to 10 million tokens – roughly a few tens of thousands of pages. You do not want to send all of that to the AI with every single question. With RAG, this happens instead:
| Step | Volume |
|---|---|
| Body of knowledge in total | 10,000,000 tokens |
| of which looked up | 5 to 20 sections |
| what actually goes to the AI | around 5,000 tokens |
Orders of magnitude for illustration, not measured values. The effect of this is manifold:
- Lower costs, because only a fraction is processed with each question – and you pay per volume processed.
- Faster answers, because a few sections are read in seconds.
- Better answers, because less irrelevant material is in the way. An AI that reads five fitting paragraphs answers more precisely than one skimming ten thousand pages.
- Any size of archive, because the volume in storage no longer limits what fits into a single request.
What this gives you day to day
- Answers from your own knowledge, not from the internet. The assistant knows your price list, your contracts, your work instructions.
- Considerably fewer invented answers. An AI invents above all where it knows nothing. If the right passage is in front of it, that risk drops substantially (Wikipedia, Atlan).
- Traceability. Good solutions name the passage they used. That is the difference between “nice” and “dependable”.
- Always up to date. A new document can be found straight away – depending on the method immediately or once it has been read in. The AI model itself never has to be touched for this.
- Permissions remain possible. Every section can carry access rights, so that accounting documents do not turn up in the workshop’s assistant (SitePoint, KeepMyPrompts).
Where the limits are
RAG does not make mistakes impossible. If the lookup finds the wrong passage, the answer sounds confident and is wrong all the same. That is why people stay part of the process: anyone who opens up the source named in the answer notices immediately whether it holds. That is precisely what the source reference is for.
Nor does RAG replace well-kept documents. An assistant that reliably finds outdated work instructions is no help at all.
What comes next
If you would like to talk through whether your own documents are suited to this: we look forward to hearing from you.
Sources
- Retrieval-augmented generation (Wikipedia) · What Is RAG? (Atlan) – definition, how it works, reduction of invented answers
- Long Context vs RAG (SitePoint) · 1M Context Windows Are a Trap (KeepMyPrompts) – selective context, costs, permissions