How does text become numbers inside the model?

Text becomes numbers in two steps, and neither of them is a neural network. First a tokenizer splits the sentence into pieces from a closed vocabulary and returns each piece’s position in that vocabulary, a list of integers. Then each integer indexes a row of a large matrix, and that row is the vector entering the first layer. The same text always comes out as the same numbers.

This is the first stretch of the journey laid out in how an LLM works inside, and the only stretch where text still exists. After the embedding matrix there are no words, letters or spaces left, only vectors. What the matrix does with the id, and what happens on the way back out, is in embedding and unembedding.

If your question is still what counts as a token, start there; and the comparison between the tokenizers in use today, with what each one charges per language, lives in tokenizers.

The path, with real values

The numbers in the figure are real, measured with cl100k_base, a public vocabulary of 100,256 entries. “The model reads numbers.” becomes five pieces:

"The" " model" " reads" " numbers" "."

The first thing to notice is that the space rides along with the piece that follows it. " model" with a leading space is a different entry from "model" without one, and the two sit at different positions in the vocabulary. That is why a prompt pasted with odd spacing sometimes behaves differently from the one you thought you sent.

The second is what happens to the same sentence in Portuguese. “O modelo lê números.” becomes six pieces, and lê, a two-letter word, splits into " l" and "ê" while " números", eight letters and an accent, stays whole. Length decides nothing. Frequency in the training corpus decides everything.

The ids here are 791, 1646, 16181, 5219 and 13. They are addresses, not measurements: id 791 is not “smaller” than 16181 in any useful sense, and neighbouring ids are usually unrelated pieces. All the meaning lives in the matrix row each id addresses. A low id does tell you one thing, though, and only one: that piece was merged early, so it was frequent in the corpus the vocabulary was built from.

There is a pre-tokenizer before BPE

The most forgotten stage of the path is the first one. Before any merging, a regular expression cuts the text into pieces the merge algorithm will never be allowed to cross. It decides, among other things, that a leading space belongs to the word that follows, that punctuation groups with punctuation, and that runs of digits get chopped into groups of at most three.

That last detail has a direct consequence. 1234567 does not reach the model as seven digits, and not as a number either: it arrives as 123, 456 and 7. The cut runs left to right, which is exactly the opposite of how a human groups thousands. The model receives a decomposition that corresponds to the place value of nothing.

It is also worth recording what unit all of this operates on. Modern tokenizers work over bytes, not characters. An accented letter takes two bytes in UTF-8, an emoji takes four, and the merge algorithm runs at that layer. That is how ê can end up as its own symbol while the syllable containing it has no entry: the merge joining the two bytes of ê was frequent enough, and the one that would join l to ê was not.

The pre-tokenizer also explains why two visually identical prompts can produce different counts. One extra line break, two spaces where there was one, an invisible character pasted from an editor: all of that changes the cut before any merge happens, and therefore changes the list of ids.

How the vocabulary is learned

The vocabulary is built once, before training, and never changes afterwards. The most common algorithm is byte pair encoding, and it is simpler than the name suggests: start from the text split into bytes, count every adjacent pair, merge the most frequent pair into a new symbol, record that merge in an ordered list, and repeat until the list reaches the size you asked for.

The idea came from machine translation, where rare words broke the model. Sennrich and colleagues proposed representing a rare word as a sequence of smaller units rather than falling back on a generic “unknown” symbol¹. Since raw bytes remain in the worst case, nothing falls outside, and the unknown symbol stops existing.

Tokenizing runs the same process in reverse: apply the learned merges, in the order they were learned. That is why the result is deterministic, and why the merge list matters as much as the symbol list — without it the vocabulary is useless.

There is an alternative family. Kudo proposed training a unigram language model over subwords and picking the segmentation by probability, which opens the door to sampling different segmentations of the same word during training as a form of regularisation, with consistent gains in low-resource and out-of-domain settings². SentencePiece, from the same author with Richardson, packaged that into a tokenizer working directly on raw text without relying on whitespace splitting, which was the precondition for training on languages that do not separate words with spaces³.

Vocabulary size

Picking the size is a trade, and both sides of it can be measured.

On the benefit side, I tokenized the three pillar articles of this site, in their Portuguese and English versions, with three public vocabularies of 50,257, 100,256 and 199,998 entries. The Portuguese texts come to 17,617, 13,359 and 11,646 tokens; the English ones to 11,390, 10,400 and 10,314. Quadrupling the vocabulary cut 34% of the tokens in Portuguese and 9% in English.

The ratio between the two languages tells the same story from another angle: with the 50k vocabulary, Portuguese costs 55% more than English to say the same thing; with the 200k one, 13% more. The larger vocabulary did not get better in general. It got better for whoever was being served badly.

On the cost side, the embedding matrix holds one row per vocabulary entry. With vectors of 4,096 numbers, the three sizes give 206, 411 and 819 million weights. In a 70-billion-parameter model that is noise; in a 1-billion-parameter model, the vocabulary becomes a large slice of the budget.

Tao and colleagues formalised roughly that calculation in 2024. Training models from 33 million to 3 billion parameters with varying vocabularies, they concluded that the optimal size grows with the compute budget and that most models run vocabularies that are too small: by their prediction, Llama2-70B should have had at least 216,000 entries instead of 32,000. Empirically, moving from 32,000 to 43,000 improved ARC-Challenge from 29.1 to 32.0 at the same FLOPs budget⁴.

Where this fails

Counting letters becomes impossible. The model never receives the letters separately. In cl100k_base, strawberry arrives as str, aw and berry, with one r closing the first piece and two inside the third. Asking how many r’s it has means asking for an inspection of something absent from the input, and what comes back is a memorised pattern about the word rather than a count.

Arithmetic inherits the cut. Singh and Strouse went after this directly and showed that the choice of number tokenization changes performance: separating numbers with commas at inference time, which forces right-to-left grouping, improved arithmetic results considerably. They also noted that errors follow stereotyped patterns, a sign that the computation is systematic rather than approximate, and that the gap between the two directions narrows as models scale⁵.

Per-language cost is uneven, and not by a little. Petrov and colleagues measured the same text translated into several languages and found token-length differences of up to 15 times, including in tokenizers deliberately trained for multilingual support; character- and byte-level models still showed over 4 times the difference on some pairs⁶. Since billing is per token, that turns into price: Ahia and colleagues analysed 22 typologically diverse languages against a commercial API and showed that speakers of many supported languages pay more for worse results, and tend to come from regions where the API was already less affordable⁷.

The inequality also shows up word by word, and it is easy to check. Still in cl100k_base, correspondence is one token and correspondência is two; internationalization is two and internacionalização is three. Length is not the reason: the Portuguese word has one letter fewer than its English cognate. The English form appeared often enough in the corpus to earn a vocabulary entry and the Portuguese one did not. Each of those words costs 50% more, every time it appears, forever.

A bad tokenizer costs quality, not just money. Rust and colleagues isolated the variable by training monolingual models on the same data, some with a monolingual tokenizer and some with a multilingual one, and concluded that the dedicated tokenizer weighs as much as the size of the pretraining corpus; replacing the multilingual tokenizer with the specialised one improved downstream performance on almost every task and language they tested⁸.

The tokenizer is frozen. Every id addresses one specific row of the embedding matrix, and that row was learned during pretraining. Swapping the tokenizer scrambles the addressing and invalidates the entire matrix. It is the one design decision in a model project you cannot revisit without retraining, which is why a badly sized vocabulary follows a model until retirement.

Nothing is unknown, and that has a price. Since raw bytes remain, any input is representable. A rare emoji, a mathematical symbol or a script thin on the ground in the corpus will not error out: they turn into a long sequence of one-byte pieces. The text goes through, and so does the bill.

What to do with this

1. Measure in tokens, with the tokenizer of the model you actually use. Character count divided by four is a guess that misses by 30% on Portuguese.
2. Do not ask the model to count characters. Count in your code and hand it the number if it needs one.
3. Separate thousands with commas when the task is arithmetic. It is a one-line prompt change and it addresses the cause.
4. Budget non-English text with headroom over the English estimate. How much depends on the vocabulary: roughly 50% more on the smaller ones, roughly 15% on the larger ones.
5. Recount when you change models. The same text changes size when the tokenizer changes, and window limits and pricing are counted at its output.
6. Stabilise the prompt prefix down to the character. One extra leading space changes the id list and drops the prompt cache.

Footnotes

1. Sennrich et al. (2016) proposed representing rare words as sequences of smaller units instead of an “unknown” symbol, with the vocabulary built by successively merging the corpus’s most frequent symbol pair. ↩

2. Kudo (2018) treated subword segmentation as inherently ambiguous and proposed sampling different segmentations during training, with a unigram model choosing between candidates; the gain showed up mostly in low-resource and out-of-domain settings. ↩

3. Kudo and Richardson (2018) packaged tokenization and detokenization into a component working directly on raw text, without relying on whitespace splitting, which allows training the vocabulary on any language. ↩

4. Tao et al. (2024) trained models from 33M to 3B parameters with varying vocabularies and concluded that the optimal size grows with the compute budget, predicting at least 216,000 entries for a model the size of Llama2-70B. ↩

5. Singh and Strouse (2024) compared left-to-right and right-to-left number tokenization on arithmetic tasks and found a meaningful gain for the latter, with stereotyped error patterns in both. ↩

6. Petrov et al. (2023) measured the same text translated into several languages and found token-length differences of up to 15 times, with over 4 times the difference even in character- and byte-level models. ↩

7. Ahia et al. (2023) analysed the cost and utility of a commercial API across 22 typologically diverse languages and showed that speakers of many of them pay more for worse results. ↩

8. Rust et al. (2021) separated the effect of the tokenizer from the effect of data size and concluded that a dedicated monolingual tokenizer weighs as much as the pretraining corpus in final performance. ↩