What is RAG (retrieval-augmented generation)?

RAG (retrieval-augmented generation) means searching your own corpus for relevant passages and sending them to the model alongside the question. The answer comes out of the text you supplied, not out of what the model memorised during training. There are two phases: an indexing pass that runs beforehand, and a search that runs on every question. It answers questions about documents the model has never seen.

The name is literal: generation augmented by retrieval. What gets augmented is the prompt. You add a handful of passages that search judged relevant, and the thing doing the adding is your own code. That assembly is the whole of RAG architecture: split documents into chunks, turn each chunk into an embedding, store, search, assemble.

None of this touches the model’s weights. The model on the other end has no idea where the text came from; to it, that’s just more context.

The two phases

Indexing runs before any question exists. You read the documents, cut each one into pieces of a few hundred tokens, run every piece through an embedding model, and store the resulting vector next to the original text and whatever metadata matters: source file, date, permissions, section. It runs once, and runs again when a document changes.

Querying runs on every question. You turn the question into a vector using the same embedding model, find the nearest vectors in the index, take the best five or ten, paste their text into a prompt with the question, and send it to the model.

One detail catches almost everyone on their first build: both phases have to use the same embedding model. A vector produced by one model is not comparable to a vector produced by another, even when both have 1,536 dimensions. Swapping the embedding model means reindexing everything, which makes it the most expensive choice to reverse in the entire pipeline.

The index is the joint between the two phases. That’s why debugging RAG is irritating: a chunk cut badly on Monday becomes a wrong answer on Friday, and the symptom shows up three stages away from its cause.

What retrieval changes about the answer

Ask a model what the response SLA is on your Pro plan. With nothing else, the answer arrives in the right shape, with the right confidence, carrying a number it reconstructed from every SaaS contract it has ever seen. “24 hours” is a plausible answer to that question in general. It just isn’t yours.

Put the paragraph from the contract in the prompt and the model reads “four business hours” and says that. It didn’t get smarter. It stopped having to guess.

The paper that coined the term measured this in 2020. Lewis and colleagues paired a seq2seq generator with a dense Wikipedia index and beat the generator alone on three open-domain question answering tasks; the generated text also came out more specific and more factual than the index-free baseline¹. The word they used for the difference was memory: the model has parametric memory in its weights, and the index is non-parametric memory that you swap without retraining anything.

The 2020 paper and what “RAG” came to mean

These two are worth separating, because the name stuck while the recipe changed.

The original work trained the whole thing. The generator was fine-tuned together with the retriever, and the retriever came from another paper that same year, DPR, which had just shown that dense retrieval trained on question-passage pairs beats BM25 by 9 to 19 absolute points of top-20 retrieval accuracy². That result is what made betting everything on vectors reasonable.

What people call RAG in 2026 almost never involves training. You call an embedding API, park the vectors somewhere, search, and concatenate strings. It’s simpler, it works well enough, and it inherited the name of an architecture that did something else. The confusion shows up when somebody reads the paper expecting to find the system they built last week.

The map of the subject

Every box is a decision with a measurable consequence, and every box is an article in this cluster.

Indexing decides chunk size and embedding model. It’s the most expensive stage to redo, because changing either one forces you to reprocess the whole corpus.

Search decides whether you go vectors only, keywords only, or both. Reranking decides how many candidates survive that search and in what order they reach the prompt.

Generation decides what the final prompt tells the model to do when the retrieved passage doesn’t answer the question. It’s ordinary prompt engineering applied to context you assembled yourself.

Evaluation is what tells the other four apart when the result is bad. It’s the stage most teams build last, after having already poked at the others in the dark.

RAG, fine-tuning, or just a prompt?

The three techniques put information in three different places, and teams tend to pick between them far too early.

The prompt holds what fits in an instruction: format, tone, a short rule. The weights hold behaviour, which is what fine-tuning changes: how to answer, in what register, with what vocabulary. The index holds knowledge that moves, and it’s the only one of the three where updating means writing a document instead of running a training job.

In practice they combine more than they compete, and the deciding criterion is how often the thing you’re trying to teach changes. Which one to use when is the question every team asks in week one, usually without the numbers to answer it.

Long context didn’t kill RAG, it moved where RAG pays off

This is the argument of the moment and it deserves to be handled without cheerleading.

The case for long context has evidence behind it. A 2024 study compared RAG against long context across several public datasets with three models of that generation, and found that when resourced sufficiently, long context wins on average performance. RAG’s advantage in that same study was cost, not quality. The authors proposed routing each query down one of the two paths, which landed close to long-context performance while paying far less³.

The case for RAG has evidence too. Models don’t use the window evenly: a 2023 paper showed performance is highest when the relevant information sits at the beginning or the end of the context and degrades significantly when the model has to reach for something in the middle, including in models marketed as long-context⁴. Pasting everything isn’t free. It’s a bet that the model will find the needle, and where the needle sits matters.

Then there’s the arithmetic, which settles most cases before the argument starts. Ten thousand pages of documentation is something like 6 million tokens. As of July 2026, the largest production context windows are around the million mark. A corpus that fits in the window is a small corpus, and most corporate corpora aren’t small.

What genuinely changed: for a small, stable corpus, a 40-page manual, an internal policy, pasting the whole text became the right answer. It’s simpler to build, there’s no index to maintain and no retrieval to get wrong. RAG stopped being the default answer to “the model needs to know about this document” and became the answer to “the corpus is too big to paste, changes often, or I need to know which passage the answer came from”. That line moves as token prices fall, and prompt caching pushes it further toward long context for stable corpora. It isn’t a fixed boundary and nobody should be selling it as one.

Where it breaks

Retrieval misses and nobody notices. If the right passage never came back, the model answers from whatever did come back, and answers well. The output looks exactly like a correct answer. This is RAG’s number one failure mode and the reason you have to measure retrieval separately from generation.

A near-miss passage is worse than a random one. A 2024 study found the opposite of what you’d expect: top-ranked passages that don’t contain the answer hurt the model, while adding random documents improved accuracy by as much as 35% in what they tested⁵. That’s one specific question-answering setup, not a general law, but the practical implication holds: raising top-k “to be safe” can make the system worse, because what you let in is precisely the almost-relevant.

Chunks cut in the wrong place. A table split down the middle, a clause divorced from its exception, a list item without the heading that said what the list was about. Retrieval works, the passage arrives, and the passage is useless.

The index goes stale. The document was updated, the index wasn’t. Freshness is RAG’s main selling point and it’s the first thing to break in production, because reindexing is a job nobody watches until the day it fails quietly.

Permissions vanish along the way. The index flattens whatever access structure the documents had. If you don’t carry permissions into the metadata and filter on them at search time, RAG becomes a data leak with a chat interface.

The model can ignore the passage. Nothing forces it to use what you handed over. When the retrieved text contradicts what it learned in training, the outcome depends on which of the two signals is stronger, and there’s no guarantee it’s yours.

What it costs

Order-of-magnitude numbers, enough to see the shape of the bill.

Indexing is a one-off. Those 10,000 pages, at 300 tokens per chunk, come to roughly 20,000 chunks. Embedding is the cheapest call in the catalogue, an order of magnitude below generation, and you pay for it when documents change, not when somebody asks a question.

Storage is smaller than people assume. Twenty thousand vectors at 1,536 dimensions in float32 take about 120 MB. That fits inside Postgres with room to spare, which is why so many teams stand up a dedicated vector database long before they need one.

Querying is the cost that repeats. Five passages of 400 tokens is 2,000 tokens of context per question. Pasting 200,000 tokens on every question costs a hundred times that, in money and in latency. The ratio between those two numbers is what decides, not the absolute price, which changes every time somebody ships a new model.

Where to start

1. Try it without RAG first. Paste the document into the prompt. If the corpus fits in the window and doesn’t move much, you’re done and you saved yourself a system.
2. Split on structure before splitting on size. Section, clause, function. Cut by character count only where there’s no structure to cut on.
3. Start with plain vector search and top-k of 5. Rerankers and hybrid search solve problems you don’t have yet.
4. Write 30 real questions with their answers and the passage that should have been retrieved. That’s the whole instrument. Without it, every change to the pipeline is a guess.
5. Measure retrieval before you measure the answer. If the right passage isn’t among those retrieved, your problem isn’t the prompt.
6. Tell the model to cite the passage and to say when it found nothing. Abstention has to be written into the prompt; it doesn’t happen on its own.

Step 4 is what separates teams that improve a RAG system from teams that swap frameworks every fortnight hoping the next one sorts it out.

Footnotes

1. Lewis et al. (2020) coined the term and measured the effect: the model with an index beat the generator alone on three open-domain question answering tasks and produced more specific, more factual text. ↩

2. Karpukhin et al. (2020) showed that dense retrieval trained on question-passage pairs beats BM25 by 9 to 19 absolute points of top-20 retrieval accuracy. ↩

3. Li et al. (2024) compared RAG against long context on public datasets: when resourced sufficiently, long context wins on average performance and RAG wins on cost. Routing each query between the two lands close to long-context quality at a fraction of the price. ↩

4. Liu et al. (2023) measured how performance drops when the relevant information sits in the middle of the context, including in models marketed as long-context. ↩

5. Cuconasu et al. (2024) found that highly ranked passages without the answer hurt the model, and that including random documents improved accuracy by up to 35% in their experiments. ↩