RAG explained, part 3: when does the effort pay off?

The central question is a different one: does the AI reliably find the right passage without having to read large parts of your material? As long as the answer is “yes”, you usually do not need RAG. How you notice that this is changing is what this article is about – using two filing systems that hold the same knowledge and are still worlds apart to work with.

Why the question of volume misleads

One reason for this is new: modern AI models read a remarkable amount at once. Current top models process around one million tokens per request (Anthropic) – that is roughly 1,500 pages.

Whatever fits into that window you can, in principle, simply send along. So as a starting point: for small and medium-sized collections you usually do not need RAG. “Usually”, because volume is not the only factor: cost per request, access rights and response time can argue against it even with small collections. We will come back to that below.

But the number alone does not carry you far. Two collections of the same size can be completely different to work with.

The factor that really counts: how well ordered is your material?

Picture two filing systems side by side that hold the same knowledge:

Diagram “the same knowledge, completely different effort” with two cards side by side. On the left “ordered – one topic per file, meaningful names” with a directory tree: dokumentation/ containing zutrittskontrolle.md, abrechnung.md, datensicherung.md and a subfolder entscheidungen/ with 001-postgres.md and 002-redis.md; below it “looking things up is almost trivial. No RAG needed.” On the right “grown over time – everything is in there, nothing can be found” with protokolle/ 8,000 meeting notes, chatverlaeufe/ years of messages, support/ 80,000 enquiries; below it “the answer sits in the middle of a long text. This is where retrieval helps.” Footer: 2,000 clean subject documents are easier to handle than 100 long sets of minutes. — Both systems hold the same knowledge. It is the one on the right that creates the work for retrieval. (Diagram in German.)

In the ordered system looking things up is almost trivial: the file name alone tells you where something is. An AI with access to the files finds the right one without you building any technology around it. In the one that has grown over time, the same answer sits somewhere in a paragraph in the middle of a long set of minutes, with nothing in a file name to hint at it. This is exactly what retrieval is made for.

The difference has a name: information density. Two thousand small, cleanly named subject documents are easier for an AI to handle than a hundred long, unstructured sets of meeting minutes – even though the second collection looks far smaller by file count.

A practical rule of thumb

What gets counted here is documents, not data volume:

Diagram “when does RAG pay off?”, marked as a rule of experience, counted in documents rather than megabytes. Four tiers below one another: 10–50 well-structured files – no RAG, send the documents along directly; 50–200 files – structure and search suffice, meaningful names, one topic per file; 200–1,000 files – depends on the content, ordered subject documents yes, long minutes rather not; from a few thousand documents – RAG becomes interesting, from 10,000 very probably needed. Footer: rule of experience, not technical thresholds – the row is an indication, not a verdict. — Counted in documents rather than megabytes – and the tier on its own settles nothing as long as the degree of order does not match it. (Diagram in German.)

Broadly speaking: below around 200 well-structured files the effort rarely pays off, from a few thousand documents retrieval becomes interesting, and from 10,000 it is very probably needed. These are not technical thresholds but a rule of experience.

Signals that argue for RAG

More telling than any tier is whether several of these points apply to you. The list is not exhaustive; it names common signals:

A lot of long, unstructured text – minutes, chat logs, support enquiries, email threads.
Knowledge sits in several places and nobody is sure where the current version is.
The material changes frequently, so keeping it up by hand cannot keep pace.
People ask by paraphrasing, not with fixed terms (see part 2).
Filters matter – this project only, this year only, only what this employee is allowed to see.
Ordinary search returns too many hits, or the AI noticeably spends time and money just finding the information in the first place.

The last signal is the most reliable one. If you observe that the AI regularly fails to find the right passage – then the moment has come.

As long as the right passage is found without any technology, RAG is a solution to a problem you do not have. Only when searching itself becomes the bottleneck does the effort pay off.

Questions of economics

Even when the volume argues for RAG, it is the usage that decides whether it pays. Four questions tend to be the most productive – depending on your business, others come on top, such as requirements from your industry or certification:

How often are questions asked? Sending the whole collection with every question costs money – every time. At ten questions a day nobody notices; at a thousand they do (SitePoint, KeepMyPrompts).
Is the material growing? An archive that grows month by month will eventually outgrow any context window.
Is everyone allowed to see everything? If not, there is hardly a way around a solution with rights management.
How fast does it have to be? Five passages found on purpose are processed in seconds; a thousand pages are not.

Our recommendation: tidy up first, then upgrade

The most effective first step is rarely a technical one. Before you start a retrieval project, the simpler exercise is almost always worth it:

Create structure – meaningful file names, one topic per document, clear folders.
Set up a signpost that says on a single page where which knowledge lives.
Work with it and watch where things get stuck.
Only then add retrieval – where searching is demonstrably the problem.

It is worth noting that professional AI tools work to this very pattern themselves: a compact overview file as the way in, with detailed topic files loaded when needed (Anthropic: Claude Code Memory).

A tidy collection makes every later RAG project better – and sometimes unnecessary.

What comes next

The next part is about the tools: what is out there, what of it is free of charge and which items still tie up money.

Sources

Anthropic model overview – context windows of current models (1 million tokens)
Long Context vs RAG (SitePoint) · 1M Context Windows Are a Trap (KeepMyPrompts) – cost per request, latency, permissions
Claude Code: How Claude remembers your project – compact overview file as the way in, detail files when needed
The document rule of thumb and the list of signals follow an engineering assessment from practice (research by Wolfgang Duttlinger, 14 August 2026) – deliberately marked as a rule of experience, not as a measured value.