What is an embedding?

An embedding is the list of numbers a model produces for a piece of text, placed so that texts with similar meaning land near each other. In a search, that is what lets the question “how do I cancel?” find the paragraph about early termination, which shares no word with it. The vector does not hold the text. It holds a position.

It is the third piece of a RAG system and the first one you don’t write yourself. You cut the document into chunks, send each chunk to an embedding model, and store the vector that comes back. At question time the same model turns the question into a vector and the system looks for the nearest ones. Everything downstream operates on those numbers and never looks at the text again.

Two neighbouring decisions get confused with this one all the time. Where the vectors live and how they get scanned belongs to vector databases; which formula measures the distance between two of them belongs to similarity metrics. The question here comes before both: where the vector comes from, what it carries, and what it drops on the way.

What “near” actually means

No axis in the space has a name. Dimension 412 is not “formality” or “financial topic”. It is whatever training put there, and there is no reason for it to be readable by a person. A number on its own says nothing. What says something is where one vector sits relative to the other vectors produced by the same model. It is also why a vector from one model never compares to a vector from another, identical dimensionality or not: each space carries its own arbitrary orientation.

Nearness is also not a natural property of text. It was trained. A modern embedding model learns from pairs somebody declared related: question and answer, title and body, a sentence and its translation. The objective pulls those pairs together while pushing random pairs apart. “Near”, in the resulting space, means precisely “resembling the kind of relation that was in the training pairs”. Nothing more general than that.

The cleanest evidence that training is the decisive part, rather than the architecture, sits in the Sentence-BERT paper. Take an off-the-shelf BERT and average its token vectors, which is the obvious move, and you get a mean Spearman correlation of 54.81 across seven textual similarity datasets; use the classification token instead and you get 29.19. Both lose to averaging static GloVe vectors, which scores 61.32. Train that same BERT on pairs, in a siamese setup, and the number goes to 74.89¹. The ability to emit vectors was there from the start. What was missing was somebody telling the model what should end up near what.

CLIP does the same for an image and its caption, one shared space, same ruler: near means training saw the two together.

From text to vector

The path has four steps, and it is the third one where information disappears.

The text gets tokenised. Every token becomes a vector, and the transformer refines those vectors layer by layer, so that the vector for “bank” in a sentence about interest rates comes out different from “bank” in a sentence about a river. Up to here you have a sequence: one vector per token.

Then comes pooling, which in most models is an arithmetic mean. Three hundred vectors become one, and what you hold from there on is the passage’s centre of mass.

What survives the mean is whatever was redundant across the passage: topic, intent, entities named, register. What does not survive is whatever depended on a specific position. Word order is the easiest loss to see — “the company pays the supplier” and “the supplier pays the company” end up as neighbours. Negation is the dangerous one, because a single “not” is one token among three hundred and barely moves the mean: “the Pro plan includes 24/7 support” and “the Pro plan does not include 24/7 support” land almost on the same point. And then there is everything that is only useful verbatim: exact figures, error codes, SKUs, order references. All of it gets split into token fragments and diluted.

That pooling is a mean is a fact about the code. That negation survives badly is a repeated empirical observation in similarity evaluation, not a law with a constant attached. You can measure it on your own corpus in twenty minutes: embed ten pairs of sentences that differ only by a “not” and look at the cosine. If it comes back above 0.9, you have just found a distinction your search does not make.

Passage length interacts with all of this and belongs to chunking. What stays here is the hard limit: every embedding model has a maximum input length, and most APIs cut the overflow silently.

How many dimensions

As of July 2026, production text embedding models return vectors of 384 to 4,096 numbers, with 768, 1,024 and 1,536 the most common sizes. What dimensionality buys you is room to keep distinctions apart: with few axes, unrelated concepts get crowded into the same corner.

What it does not buy is quality. A 3,072-dimension model is not automatically better than a 768-dimension one — the training data and the objective decide that, and the gap between two models at the same size is usually wider than the gap between two sizes of one model. Dimensionality becomes a selection criterion because it is the number printed on the spec sheet, not because it predicts anything.

On the cost side the arithmetic is direct. A float32 vector takes 4 bytes per dimension: 3 KB at 768 dimensions, 12 KB at 3,072. A million chunks is therefore 3 GB or 12 GB, and that same factor shows up again in scan time, which grows linearly with dimensionality under exact search.

There is a way out of deciding this at index time. Matryoshka Representation Learning, from 2022, trains the representation so that the leading numbers of the vector already form a usable representation on their own: you truncate 3,072 to 512 and lose little, because the information was organised coarse-to-fine. Their headline result is from vision, with up to 14 times fewer dimensions at the same ImageNet-1K classification accuracy, and the technique was extended to BERT and other modalities². It is why some embedding APIs now accept a dimension parameter: underneath, that is vector truncation.

“King minus man plus woman”

This is the most-quoted demonstration about embeddings and the source of the idea that the space is a coordinate system where directions carry meaning. It comes from the word vectors of the word2vec lineage; the 2013 paper that popularised the method built the analogy test set that served as the standard evaluation for close to a decade: 8,869 semantic and 10,675 syntactic questions³. As a first intuition it earns its keep. As a mental model for someone building retrieval, it misleads.

The method behind it is literal: add the vectors, find the nearest vector to the result. The detail almost nobody repeats is that the standard implementation bars the three input words from the candidate list. A 2016 study measured what happens without that exclusion: the nearest neighbour of the result was the starting word 93% of the time, and the second term of the analogy 5% of the time⁴. The arithmetic lands, almost always, back where you set off from. “Queen” is what is left after forbidding “king”, “man” and “woman”.

The same study showed that much of the accuracy does not come from the offset at all. A baseline that ignores the offset entirely, just returning the nearest neighbour of the third term, reached 70% on the plurals category against 80% for the full method. Walking in the direction opposite to the offset still scored 45%. And accuracy ranged from 13% to 90% depending on the analogy category, which by itself says you are not measuring one single property of the space.

Three things fall out of this for anyone building retrieval. First, directions are not manipulable with anything like the reliability the demo implies; “subtract the formality direction” from a chunk vector is an idea that sounds excellent and does not survive measurement. Second, today’s embeddings are sentence and passage embeddings trained for a different objective — analogy structure is not what they optimise for. Third, what you actually hold is a ranking: the order of the neighbours is the useful output, and the absolute cosine value tells you far less than people assume.

That last point has formal backing. A 2024 paper analysed embeddings derived from regularised linear models, where closed-form solutions exist, and showed that cosine between them can yield arbitrary similarities: in some cases not even unique, in others implicitly determined by the regularisation used in training⁵. In deep models, where several regularisations combine, the authors caution against applying cosine blindly. The practical reading is short: a threshold of 0.8 does not mean the same thing across two models, or across two corpora under one model.

Choosing the model

The leaderboard settles less than it appears to. MTEB, published in 2022, benchmarked 33 models across 8 embedding tasks, 58 datasets and 112 languages, and its central finding was that no method dominated every task⁶. A model that is strong at textual similarity is not automatically strong at retrieval, which is the task you have.

Worse: in-domain performance does not predict out-of-domain performance. BEIR, from 2021, evaluated ten retrieval systems across 18 datasets and found that BM25, keyword search with no vectors anywhere, trails neural methods by 7 to 18 points inside the training domain on MS MARCO, and turns into a strong baseline outside it. DPR, a dense retriever, averaged 47.7% below BM25 across the 18 datasets. On BioASQ, which is biomedical, BM25 scored 0.465 nDCG@10 against DPR’s 0.127⁷.

That is not an argument against embeddings. It is an argument against picking a model from somebody else’s table. The order that works:

1. Filter on what is binary. The languages in your corpus, the maximum input length, the licence. That halves the list before you measure anything.
2. Measure on your own corpus. Thirty real questions, each annotated with the passage that should come back, and the metric being how often it lands in the top five. Two or three candidates, a thousand documents, an afternoon.
3. Only then look at dimensionality and price. Truncation and quantisation are cost decisions, and you cannot price them before you have a quality baseline.

One operational caveat: changing the model forces a full reindex, which makes it the most expensive decision to reverse in the pipeline. Expensive in coordination, not in money — as of July 2026, embedding is still the cheapest call in any catalogue.

Where it breaks

Similar is not relevant. A model trained on symmetric pairs puts “how do I cancel my subscription?” near “how do I cancel my plan?”, which is another question, rather than near the paragraph that answers it. The symptom is a search that returns the FAQ instead of the answer. The fix is usually an asymmetric model, or the instruction prefix some models require, of the query: and passage: kind; leaving that prefix out degrades retrieval with no warning at all.

Whatever training never saw. Order numbers, error codes, SKUs, function names: the tokeniser splits them into fragments and the mean dilutes what’s left. Legal, medical or product vocabulary falls into the same hole for a different reason, that of sitting outside the training domain. Together they are the strongest case for keeping a keyword index alongside the vector one.

The vector is not reversible. You cannot reconstruct the text from it. That seems obvious, and yet indexes reach production holding nothing but the numbers, and then there is no text to paste into the prompt.

The model version disappears from the index. A partial reindex with a different version of the same model leaves you with two spaces mixed into one, and the symptom is a search that works well for some documents and badly for others with no visible pattern. Storing the model name and version on every row costs a few bytes and is what makes the problem visible.

All the scores look alike. When the top-20 cosines sit between 0.71 and 0.74, you do not have a ranking, you have sorted noise. The cause is usually text repeated in every chunk pushing all the vectors the same way, or simply the wrong model for that domain.

What it costs

Order-of-magnitude numbers, enough to see the shape of the bill.

Indexing is one pass over the whole corpus. A million chunks at 300 tokens is 300 million tokens of embedding, paid once — and again on every change of model or of chunk boundary.

Storage is the recurring cost, and it is where dimensionality bites. A million 1,024-dimension vectors in float32 is 4 GB; the same vectors in int8 is 1 GB. Quantisation is the cheapest cost lever in the pipeline and is almost always pulled before anybody has measured what it does to quality.

Querying is negligible on the embedding side. A question is a few dozen tokens, and the per-question cost lives in generation, not here.

Footnotes

1. Reimers and Gurevych (2019) measured that averaging token vectors from a BERT with no pair training scores below averaged static GloVe vectors on textual similarity, and that siamese training on pairs reverses it. ↩

2. Kusupati et al. (2022) trained nested representations in which a prefix of the vector works on its own, with up to 14 times fewer dimensions at the same ImageNet-1K classification accuracy. ↩

3. Mikolov et al. (2013) proposed the architectures that became known as word2vec and assembled the analogy set of 8,869 semantic and 10,675 syntactic questions used to measure linear regularity in the space. ↩

4. Linzen (2016) showed the offset method depends on excluding the input words: without that, the nearest neighbour of the result is the third term 93% of the time. A baseline that ignores the offset reached 70% on plurals. ↩

5. Steck et al. (2024) derived analytically that cosine between embeddings from regularised linear models can be arbitrary, and that in deep models the value is shaped by regularisations nobody chose with similarity in mind. ↩

6. Muennighoff et al. (2022) benchmarked 33 models across 8 tasks and 58 datasets and found no method that dominated all of them. ↩

7. Thakur et al. (2021) showed in-domain performance does not predict generalisation: BM25 trails by 7 to 18 points on MS MARCO and is a strong baseline on the 18 BEIR datasets, where DPR averaged 47.7% below it. ↩