How do you build a RAG system from scratch?

A RAG system has seven parts: load the document, cut it into pieces, turn each piece into a vector, store the vectors, search, assemble the prompt, generate. None of them needs a framework. The core fits in about forty lines of Python with a file on disk standing in for the database. Four of those seven decide how good the answers are.

This article is about the build. If the question is still what the acronym means and when it pays off, that lives in what is RAG. If the question is where to cut the text, chunking covers only that — it is the part with the most consequence and the most folklore attached.

The architecture you are about to build is not the one from the 2020 paper. The original work fine-tuned the generator together with the retriever¹; what people build today is a pipeline of API calls stitched together by your own code. It kept the name and swapped the recipe. And the part almost nobody sees coming on a first build is how much of the result rides on the metadata travelling alongside the vector.

The seven parts

1. Loading means getting text out of a file. In Markdown that is reading the file; in PDF it is a whole problem, with column reading order, tables, and the page header that repeats itself into every chunk. This is where the work gets underestimated most often.

2. Splitting turns a document into a list of pieces. Each piece becomes a unit of retrieval: it is what search hands back and what the model reads. A cut that separates a clause from its exception produces an answer that is technically in the text and still wrong.

3. Embedding runs each piece through an embedding model and gets back a vector of a few hundred or a few thousand numbers. It is the only call in the pipeline you pay for by size of corpus rather than by number of questions.

4. Indexing is storing the vector next to the original text and the metadata: source file, section, date, permissions. Storing the vector alone works right up until the first time somebody asks where an answer came from.

5. Searching embeds the question with the same model and looks for the nearest vectors. “Same model” is not a detail: vectors from different models are not comparable, even at identical dimensions.

6. Assembling concatenates the retrieved passages, the question and the instruction into one prompt. It is the cheapest part to change and the most ignored.

7. Generating is the model call. It is the only stage most teams watch, and the one that explains the result least.

The core, in forty lines

Two functions are left out because they depend on your provider: embed, which returns a vector for a piece of text, and generate, which returns text for a prompt. Everything else is this:

import glob, json
import numpy as np

def split(text, size=1200, overlap=200):
    parts, i = [], 0
    while i < len(text):
        parts.append(text[i:i + size])
        i += size - overlap
    return parts

docs = []
for path in glob.glob("corpus/**/*.md", recursive=True):
    for n, piece in enumerate(split(open(path).read())):
        docs.append({"file": path, "n": n, "text": piece})

V = np.array([embed(d["text"]) for d in docs], dtype="float32")
V /= np.linalg.norm(V, axis=1, keepdims=True)
np.save("index.npy", V)
json.dump(docs, open("index.json", "w"))

def search(question, k=5):
    q = np.array(embed(question), dtype="float32")
    q /= np.linalg.norm(q)
    return [docs[i] for i in np.argsort(-(V @ q))[:k]]

def answer(question):
    passages = search(question)
    context = "\n\n".join(f"[{d['file']}#{d['n']}]\n{d['text']}" for d in passages)
    return generate(
        "Answer using only the passages below. Cite the bracketed identifier "
        "next to every claim. If the passages do not answer the question, say "
        "you could not find it.\n\n"
        f"{context}\n\nQuestion: {question}"
    )

The 1,200 characters in the split come to roughly 300 tokens of English prose. Normalising before np.save is what lets a dot product stand in for cosine similarity: at unit length the two give the same number, and V @ q compares the question against the entire corpus in a single matrix multiply.

That is a complete RAG system. Everything production demands is missing — batching the embedding calls, retries, incremental reindexing of what changed, a permission filter before the argsort — and none of it changes the architecture. Those are layers around the same seven parts.

The four decisions that set the quality

Where the text gets cut. Cutting by character count, as the example above does, is the worst option and the right one to start with: it is predictable, and it gives you the baseline against which to measure whether cutting by section actually helps. Cut on structure wherever structure exists — heading, clause, function — and fall back to size only for what is left.

Which embedding model. This is the most expensive decision to reverse, because swapping the model invalidates the whole index. Spend half an hour comparing two or three before you index 200,000 chunks, and record the model name and version inside the index file. An index without that stamp is a time bomb: one day somebody swaps the model, nothing in the pipeline complains, and retrieval quietly starts returning noise.

How many passages, in what order. Five is a reasonable start. Pushing to twenty “to be safe” usually makes things worse, because what you let in is the near-miss — the passage that discusses the right topic without containing the answer. A 2024 study that swept combinations of pipeline stages measured exactly this trade between performance and efficiency, and found a point past which adding retrieval stops paying².

Order is the forgotten half of that decision. argsort hands back passages from nearest to farthest and most pipelines concatenate in that order without thinking about it. But models do not read the context evenly: a 2023 paper moved the relevant information across positions in the prompt and found performance drops sharply when it sits in the middle, including in models marketed as long-context³. With five passages this barely matters. With twenty, your fifth-best landed precisely where the model looks least.

What the prompt says when nothing good came back. If the instruction never covers the empty-retrieval case, the model fills it in. That is one line of text, and it is the difference between “I could not find that in the corpus” and a fabricated number wearing the same face as a correct one.

The first two force you to reprocess the corpus. The last two you change between coffees. That does not make the cheap ones less important — it means you should exhaust the cheap ones before you touch the expensive ones.

When the vector file stops being enough

The linear search in the example compares the question against every vector. Fifty thousand chunks at 1,536 dimensions is 77 million multiplications per question, which in numpy lands in the tens of milliseconds on a laptop — order of magnitude, not a measurement. For an internal corpus with dozens of users, that conversation is over.

The tipping point is usually not latency. It is process memory, metadata filtering and concurrent writes. A matrix of 500,000 float32 vectors takes about 3 GB, and at that size loading it into every worker stops making sense.

When that day comes, what replaces it is an approximate index. HNSW, the algorithm behind a good share of today’s vector databases, builds a layered proximity graph and searches from the top layer down, with a cost that scales logarithmically rather than linearly⁴. “Approximate” is literal: you trade recall for speed, and the knob controlling that trade has to be measured against your own questions rather than left at the default.

Worth noting that dense search is neither the only option nor always the best one on its own. DPR, the 2020 work that made betting on vectors reasonable, beat BM25 by 9 to 19 absolute points of top-20 accuracy⁵ — but that was measured on open-domain questions phrased in natural language. On a corpus full of part numbers, standards references and proper nouns, keyword search tends to beat vectors on the exact-match cases, and combining the two is what actually works.

Where it breaks

A 2024 experience report followed three RAG systems in production — in research, education and biomedicine — and catalogued seven recurring failure points. The two conclusions the authors highlighted are worth more than the list itself: validating a RAG system is only feasible while it is running, and its robustness evolves rather than being designed in at the start⁶. Anyone hoping to settle the architecture on paper before seeing real questions is solving the wrong problem.

What that means in practice is that RAG failures have no log. They arrive as well-written answers:

The cut separated the clause from its exception. The retrieved passage is correct and incomplete. The answer states the rule and drops the “except when”, and nothing in the system knows anything is missing.

Retrieval never returned the right passage. The model answers from whatever did come back, and answers well. This is the failure mode that keeps a RAG system hallucinating even with the correct corpus indexed.

The index went stale. The document changed, reindexing failed quietly. Freshness is RAG’s headline argument and the first thing to break, because the job that maintains it is the one nobody watches.

Permissions vanished. The index flattens whatever access structure the files had. With no permission metadata and no filter before the top-k cut, you have built a data leak with a chat interface.

The model ignored the passage. Nothing forces it to use what you handed over. When the retrieved text contradicts what it learned in training — an internal policy that departs from the industry norm, a field name that means something else at your company — the outcome depends on which signal is stronger. There is no guarantee it is yours, and the symptom is an answer that blends both versions without saying so.

What it costs

Order of magnitude, for a support corpus of 1,200 articles, call it 3 million characters.

Indexing yields around 3,000 chunks of 300 tokens, or 900,000 tokens of embedding. It is the cheapest call in the catalogue, an order of magnitude below generation, and you pay for it when a document changes — not when somebody asks.

Storage is 3,000 vectors at 1,536 dimensions in float32: about 18 MB. It fits in a file, fits in memory, fits in Postgres. This is not where the project gets stuck.

Querying is the cost that repeats: five passages of 300 tokens is 1,500 tokens of context per question, plus the question and the instruction. The question’s embedding is rounding error. What decides viability is questions per day multiplied by context size, which is why raising top-k from 5 to 20 shows up on the invoice before it shows up in the quality.

Where to start

1. Write thirty real questions before you write any code. Each with its expected answer and the file it should have come from. Without that, every change to the pipeline is a guess.
2. Split by character count with overlap and move on. Tuning the split before retrieval works at all is tuning in the dark.
3. Store the text and the metadata beside the vector from day one. Reindexing to add a field you forgot is the most avoidable rework on this list.
4. Measure retrieval alone. Run the thirty questions and count how many had the right passage in the top five. That number is your system’s ceiling: no prompt recovers what search never returned.
5. Write the abstention instruction and the citation instruction into the same prompt. Both together, because asking for citations without allowing abstention pushes the model to cite the wrong passage.
6. Only then consider a reranker, hybrid search or a vector database. Each solves a specific problem, and you need to know which of the three you have.

Step 4 is what separates teams that improve a RAG system from teams that swap libraries every fortnight.

Footnotes

1. Lewis et al. (2020) trained generator and retriever together and called the index non-parametric memory — the architecture that popularised the term is not the one most people build today. ↩

2. Wang et al. (2024) swept combinations of RAG pipeline stages looking for the balance between performance and efficiency, and showed that adding retrieval reaches a point where it stops paying off. ↩

3. Liu et al. (2023) measured how performance falls as the relevant information moves through the context: best at the beginning and the end, worst in the middle, including in long-context models. ↩

4. Malkov and Yashunin (2016) describe HNSW, a layered proximity graph whose search cost scales logarithmically with the size of the index. ↩

5. Karpukhin et al. (2020) showed that dense retrieval trained on question-passage pairs beats BM25 by 9 to 19 absolute points of top-20 accuracy on open-domain questions. ↩

6. Barnett et al. (2024) catalogued seven failure points across three production case studies and concluded that validating a RAG system is only feasible in operation, and that robustness evolves rather than being designed in up front. ↩