What happens inside an LLM between prompt and answer?

Between the prompt and the answer the model always does the same thing, in the same order: it splits the text into tokens, turns each token into a vector, pushes those vectors through dozens of blocks that rewrite them, converts the result into a score for every token in the vocabulary, and draws one. Then it does the whole thing again, with the drawn token appended to the input.

None of that is a decision about what to do next. The sequence of operations is fixed and runs for any input; what changes is the numbers flowing through it. This is the transformer architecture, published in 2017 for machine translation¹, and the skeleton has moved very little since.

This article walks the whole path once, without maths. Each stage has its own article in the guide — the residual stream that stitches the blocks together, and the attention block up close. If the open question is still what a language model is, start there.

From text to tokens

The model never sees letters. Before any arithmetic, a tokeniser cuts the text into pieces drawn from a closed vocabulary of tens to hundreds of thousands of entries, and hands back the index of each piece. It is a lookup table, not a neural network: the same text always comes out as the same list of numbers.

That vocabulary is built once, before training, almost always with some variant of byte pair encoding. The idea came out of machine translation, where rare words broke the model: instead of a whole-word vocabulary with a catch-all “unknown” symbol, merge the most frequent character pairs in the corpus until you hit the size you want². A common word becomes one token, a rare word becomes three or four pieces, and nothing falls outside, because in the worst case you are left with raw bytes.

Three consequences you meet in practice:

• Counting letters is hard for the model. “Strawberry” may arrive as a single symbol. Asking how many r’s it has means asking the model to inspect something it never received in parts.
• The space is part of the token. “ the“ and “the” are usually different vocabulary entries, which is why a prompt pasted with odd spacing sometimes changes the answer.
• Anything but English costs more tokens. The common vocabularies were fitted on mostly-English corpora, so accents and inflection fragment into more pieces. The same paragraph translated bills more.

From token to vector

Each tokeniser id indexes a row of a large table, and that row is a vector of a few thousand numbers. This is the embedding, and it is the only point in the model where a token has a fixed representation: here the vector for “bank” is the same whether the sentence is about money or about a river.

Position is still missing. Attention on its own has no sense of order — to it, a set of vectors is a set. The 2017 paper solved that by adding a pattern of sines and cosines to the vector, one per position¹. More recent models encode position differently, inside the attention block itself, but the effect is the same: the network learns who came before whom.

The block: attention and feed-forward

From there the model applies the same block N times. N sits in the dozens, from thirty-odd in small models to more than eighty in large ones. Every block has the same shape and different weights.

Each block holds two operations, in this order: attention, which moves information between positions, and the feed-forward layer, or MLP, which works on each position in isolation.

The detail that organises everything else: neither one replaces the vector. Each computes a correction and adds it to what was already there. That vector crossing the entire model, collecting sums, is the residual stream. Think of it less as a pipe carrying information and more as a whiteboard: every block reads what is written, adds a piece, and passes it on. What the first block wrote is still readable at the last.

What attention does

At each position the model builds three vectors out of what sits in the residual stream: a query, a key and a value. The query at one position is compared against the key of every earlier position; the closer the match, the more weight. The result is an average of the values, weighted by those scores, added back into the stream.

Put plainly: the current position asks “what back there matters to me?” and pulls it forward. This is the mechanism that lets a pronoun find its antecedent thirty words earlier, and it is the only place in the block where different positions talk to each other.

Two constraints are worth noting. Attention is causal: each position only looks backwards. That is not thrift, it is what allows every position to be trained in parallel and what makes the cache further down possible. And it is split into heads, dozens of parallel copies each working in its own smaller subspace, which end up specialising in different kinds of relation.

What the feed-forward layer does

The feed-forward layer is where most of the weights live: two-thirds of a transformer’s parameters, by the count in a 2021 paper that set out to find what those layers store³. It takes one position’s vector, expands it, applies a non-linearity and comes back to the original size. No position looks at any other.

That same paper offers the most useful reading available of what this means. The first matrix behaves like a set of keys, each responding to a pattern in the text: in the lower layers, shallow patterns such as a word ending; in the upper ones, semantic patterns. The second behaves like values, and each key that fires writes a lean towards certain output tokens into the stream³. The feed-forward layer is the memory, and attention is the routing.

Normalisation, the part nobody draws

Before each of the two operations, the vector is rescaled. It looks like paperwork and it is not: without it, a stack of eighty blocks does not train.

The paper that settled the question showed that where the normalisation sits changes how gradients behave at initialisation. In the original layout, with the normalisation between blocks, gradients near the output start out large and training is only stable with a carefully tuned learning-rate warm-up. Move the normalisation inside the residual branch and they are well behaved from the start, and the warm-up can be dropped⁴. Nearly every model since has adopted that placement.

When the feed-forward layer is an MoE

If the feed-forward layer holds most of the weights, that is where there is something to save. The mixture-of-experts idea is to swap one feed-forward layer for several — the experts — and put a router in front that picks a small subset of them for each token.

Two things get misread here.

Routing is per token and per layer, not per question. There is no “code expert” handling your whole request. Two tokens in the same sentence go to different experts, and the same token can take different paths at layer 10 and at layer 40.

The saving is in arithmetic, not memory. Every expert’s weights have to be loaded; only a fraction runs per token. That is why an MoE model card carries two numbers, total and active parameters: the second predicts speed, the first predicts how much GPU you need. The Switch Transformer, which fixed the choice at one expert per token, measured up to 7 times faster pre-training on the same compute budget⁵.

The price is instability. Without explicit pressure to spread the load, the router collapses onto a handful of experts and the rest go unused.

From vector to token

After the last block, the vector at the final position carries everything the model assembled. What remains is turning it into a word.

The conversion is a multiplication against the whole vocabulary: one number per possible token, the logit. In many models this is the embedding table itself, run backwards. A normalisation then turns those numbers into a probability distribution.

And then the model draws. This is the only non-deterministic step in the path, and the only one you steer from outside without touching the prompt:

• Temperature flattens or sharpens the distribution before the draw. Near zero, the most likely token wins almost every time.
• Top-k discards everything outside the k most likely tokens.
• Top-p, or nucleus, cuts the tail by cumulative mass, so the cut is wider when the model is unsure and narrower when it is confident.

Always taking the most likely token looks like the obvious choice and is not. A 2019 paper showed that decoding by maximum likelihood produces bland, oddly repetitive text even from good models, and that truncating the less reliable tail before drawing lands much closer to the distribution of human text⁶. Output quality moves with the decoding strategy, with the model untouched.

What is computed once and what repeats

Everything above describes one pass. Generating a 500-token answer is 500 passes, and they do not cost the same: the split between prefill and decode explains most of the bill for serving a model.

Prefill is the first pass. The whole prompt goes in at once and every position is processed in parallel, because the causal mask already guarantees nobody looks forward. There is a lot of arithmetic per weight read, so the GPU runs near what it is good at. The cost grows with prompt length, and it is what sets the time until the first token appears.

What prefill leaves behind is the cache: the keys and values for every prompt position, kept.

Decode is all the rest, one token at a time. Each new token computes its own query, key and value, attends over the cache and appends its own. The arithmetic per token is small, but to do it the GPU has to read every weight in the model out of memory. That read comes to dominate the work. A 2022 study of inference on models above 500 billion parameters reached 29 ms per token at small batch sizes, and showed that shrinking the cache, by having several queries share one key, allows context up to 32 times longer⁷.

Three things follow, and they account for most of the economics of serving an LLM:

1. A long prompt costs you at the start; a long answer costs you throughout. Different levers, and confusing them means tuning the wrong side.
2. Large batches are nearly free during decode. If the weights are going to be read anyway, serving 32 requests off that read is far cheaper per token than serving one. It is the main reason serving at scale costs less per token than running the same model on your own.
3. The cache is what limits concurrency. It grows linearly with context and with the number of simultaneous requests.

An order-of-magnitude sum for the third point, with round numbers rather than a specific model: 80 layers, 8 key groups of 128 dimensions, at 16 bits. Each cached token takes 2 × 80 × 8 × 128 × 2 bytes, roughly 0.33 MB. A hundred thousand tokens of context turn into some 33 GB, cache alone, before any weights.

Where this account breaks down

It describes what gets computed. It does not describe what any of it means.

Attention weights are not an explanation. It is tempting to read an attention map as “the model looked at this word, so it answered that”. A 2019 paper tested exactly that and found the opposite: attention weights often fail to correlate with gradient-based measures of feature importance, and you can construct very different attention distributions that yield the same prediction⁸. Scope matters, since those were attention classifiers rather than today’s LLMs, but the warning holds, and the argument that followed never settled cleanly.

The skeleton is stable, the details are not. Tokeniser, embedding, blocks, unembedding and sampling are in every decoder-only model being served in 2026. The normalisation variant, how position is encoded, how many key groups attention shares, whether the feed-forward layer is dense or MoE — those change from model to model. Reading the model card is still necessary.

Temperature zero does not guarantee identical output. It removes the draw, not the numerical wobble: the same request in a different batch can sum the same terms in a different order, and a near-tie between two tokens becomes a different answer. Determinism is something to test, not to assume.

This is inference, not training. Nothing here explains where the weights came from, and the answer to “why did the model say that” lives in training far more than in the architecture.

What to do with this

1. Measure in tokens, not characters. Budget, cost and window limits are all counted on the tokeniser’s output, not in your text editor.
2. Separate the two clocks. If the problem is a slow start, work on the prompt. If it is a slow finish, work on answer length or on the model.
3. Keep the prefix stable. Prompt caching reuses the prefill of an identical prefix, which only works if the fixed part comes before the variable part.
4. Pick decoding by task. Extraction and classification want low temperature. Text that must not repeat itself wants nucleus.
5. Read both MoE numbers. Active to estimate speed, total to estimate memory.
6. Count cache memory, not only the window size. A large context under real concurrency hits the GPU before it hits the advertised limit.

From here the guide splits like this:

If you only read one more, read the one on the residual stream. It is what makes the other four make sense together.

Footnotes

1. The paper that introduced the transformer, with multi-head attention, residual connections and sinusoidal position encoding. The target was machine translation; the decoder on its own later became the basis of LLMs. ↩ ↩²

2. Sennrich et al. (2016) proposed representing rare words as sequences of smaller units instead of falling back on an “unknown” symbol, and measured gains of 1.1 and 1.3 BLEU on the WMT 15 English-German and English-Russian tasks. ↩

3. Geva et al. (2021) showed that feed-forward layers, which account for two-thirds of the parameters, operate as key-value memories: keys match textual patterns and values induce distributions over the output vocabulary. ↩ ↩²

4. Xiong et al. (2020) showed that with normalisation between blocks the gradients near the output are large at initialisation and require learning-rate warm-up; placed inside the residual branch they are well behaved and the warm-up can be removed. ↩

5. Fedus et al. (2021) simplified routing to one expert per token and measured up to 7 times faster pre-training on the same compute budget, on models derived from T5. ↩

6. Holtzman et al. (2019) showed that decoding to maximise likelihood degenerates into repetitive text, and proposed sampling from a dynamic nucleus of the distribution, truncating the less reliable tail. ↩

7. Pope et al. (2022) built an analytical model of inference cost for large transformers, reached 29 ms per token at small batch sizes on 500B+ models, and showed that multi-query attention, by storing less cache, allows context up to 32 times longer. ↩

8. Jain and Wallace (2019) tested whether attention weights explain predictions and concluded they do not: they often fail to correlate with gradient-based importance, and very different distributions produce the same output. ↩