What is the transformer architecture?

A transformer is a stack of identical blocks, and each block does two things: an attention sublayer lets every position in the text look at every other, and a feed-forward network processes each position alone. Around both sit normalisation and a sum that adds the input back. It was proposed in 2017 for machine translation and became the base of nearly every LLM.

The paper’s title states the bet: attention is enough. Until then, attention was an accessory bolted onto a recurrent network, and the 2017 contribution was removing the recurrence entirely and keeping only the accessory. The headline result was translation, and the architecture had one side that read and one that wrote — the encoder and decoder split that LLMs later abandoned, keeping only the second.

What follows describes the structure. What happens inside it when you send a prompt, from token to answer, belongs to how an LLM works, and a critical reading of the 2017 paper lives somewhere else again.

The block, opened up

The original paper describes the block in one sentence: each layer has two sublayers, multi-head self-attention and a position-wise feed-forward network, with a residual connection around each followed by normalisation¹. Each piece is worth opening.

Attention is the only part that mixes positions. Each position computes how much every other one matters to it and builds its representation out of that mix. Multi-head means doing this several times in parallel with different projections: the original model used 8 heads over a dimension of 512, 64 dimensions per head¹.

Feed-forward is the only part that mixes nothing. It is a small network applied to each position separately, with the same weights everywhere. It is also where most of the block’s parameters live: with an inner dimension of 2,048 against the model’s 512, the original paper’s arithmetic puts roughly two thirds of the block’s weights in feed-forward and one third in attention. A 2020 study argued these layers behave as key-value memories, where each key correlates with a textual pattern and the matching value induces a distribution over the output vocabulary².

The residual sum is what makes stacking possible. Each sublayer’s output is added to its input rather than replacing it. The technique comes from computer vision, in 2015, and it is what fixed very deep networks getting harder to train instead of better³. Without it, a stack of 32 blocks does not converge.

Normalisation keeps the numbers in range. The detail nobody draws is where it sits. The 2017 paper normalised after the residual sum, and that choice forces a careful learning-rate warm-up. A 2020 study showed why: with normalisation outside the residual block, gradients near the output are large at initialisation, which makes training unstable at a high rate. Move normalisation inside the block and the gradients behave, so the warm-up can be dropped⁴. Practically every current model does the latter.

Why attention replaced recurrence

A recurrent network reads the sentence word by word and carries a state from one position to the next. That creates two difficulties. The first is distance: information from the first word reaches the tenth after nine steps, degraded at each one. The second is that the steps cannot be computed out of order, so no amount of GPU helps process a sequence faster than sequentially.

Attention existed before the transformer, and it existed precisely to fix the first difficulty. A 2014 paper added to a recurrent translation model a mechanism that searched, for each generated word, the relevant parts of the source sentence, removing the need to squeeze everything into a fixed-size vector⁵. It worked, and the recurrence stayed underneath.

The 2017 move was dropping the recurrence and keeping only the search. With no state passed along, every position can be computed in the same pass, and the distance between any two positions stops mattering: one looks at the other directly.

It is worth being precise about where that gain shows up, because the misunderstanding is common. The parallelism applies to training and to reading the prompt, when the whole text already exists. It does not apply to generation: each new token depends on the previous one, and the model runs once per token. That is why the answer appears word by word on screen even in an architecture sold as parallel.

The stack, and what changes along it

The 2017 model stacked 6 blocks on each side¹. Llama 3 uses 32 blocks in the 8-billion-parameter version, 80 in the 70-billion one and 126 in the 405-billion one⁶. Each block’s structure is identical across all of them; what changes is how many and how wide.

Blocks being identical does not mean they do the same work. A 2019 study probed an encoder model layer by layer and found the steps of the classical language processing pipeline represented in order: part of speech first, then parsing, then named entities, then semantic roles, and coreference last. The authors also observed the model revising low-level decisions once higher-level information disambiguates them⁷.

Closer to the output, a 2022 study showed you can read each feed-forward layer’s update directly in vocabulary space: each one pushes the distribution toward concepts that are often human-interpretable. The authors used that for two concrete things: cutting GPT-2’s toxicity by almost half and, with a simple early exit rule, saving 20% of the computation on average⁸.

Honesty matters here. These measurements come from specific models of modest size by 2026 standards. The general ordering repeats across models; the exact cuts do not. Nobody should look at a new model and claim that layer 17 does coreference.

What has changed since 2017

The skeleton is the same and almost every part has been swapped. Four substitutions cover most of the distance.

Position. Attention knows nothing about order: to it the input is a set, not a sequence. The original paper solved that by adding sinusoids of different frequencies to the input vector¹. Since 2021 the standard has been rotating the query and key vectors by an angle proportional to position, which makes relative position fall out of attention’s inner product naturally⁹. Much of the context-window extension work operates on exactly that angle.

Where normalisation sits. After the residual sum in 2017, inside the residual block since 2020⁴.

Which normalisation. LayerNorm computes two statistics per position, a mean and a variance, then recentres and rescales. The variant that took over drops the recentring and keeps only the rescaling, which halves the statistics and turned out to be empirically equivalent on language. None of that changes what the model can do; it changes how fast a step runs, which at this scale is the same thing as changing how much model you can afford.

The feed-forward activation. ReLU in the original, gated linear unit variants in current models. The gate is an extra multiplication that lets the layer suppress its own output per dimension, and it costs a third matrix in the sublayer.

There is one more change that came from a memory constraint rather than a quality argument: head sharing. In the original design every attention head had its own keys and values. Llama 3 at 8 billion parameters has 32 query heads and only 8 key and value heads⁶, so groups of four query heads read the same stored keys and values. That does not make the model better, and it costs a little quality. What it buys is a cache four times smaller during generation, and the next section shows the size of the problem it solves.

Fixed depth, fixed effort

There is a property of the block that slips past most explanations and accounts for a lot of observable behaviour: a transformer spends exactly the same computation per token regardless of difficulty. Thirty-two blocks for “yes” and thirty-two blocks for the decisive step of a proof. There is no internal loop, no “think a bit longer” inside one pass.

Two consequences follow that you see daily. The first is that the only way for the model to spend more computation on a hard problem is to generate more tokens: that is literally what chain-of-thought buys, compute purchased in units of token. The second is that much of the depth is wasted on easy cases, and that is measurable — the 2022 feed-forward study saved 20% of the computation on average just by stopping the stack once the prediction was already settled⁸.

The caveat is that this describes the architecture, not the products. A system that routes easy questions to a smaller model is solving the same problem from the outside, with two fixed-depth models instead of one variable-depth model.

Where it breaks

The quadratic term. Attention compares every position with every other, so the work per layer grows with the square of the length. Doubling the context quadruples that part of the bill. For short sequences it does not even dominate the cost, which sits in feed-forward; attention takes over as the token count approaches the model dimension, which in Llama 3 at 8 billion is 4,096⁶.

The key-value cache. During generation the model stores the keys and values of every previous token so it does not recompute them. That cache grows linearly with context and usually runs out before compute time does. With Llama 3’s 8-billion numbers — 32 blocks, 8 key-value heads, 128 dimensions per head — that is about 128 KB per token in 16-bit, or 4 GB for 32 thousand tokens. Without head sharing it would be 16 GB, which does not fit on a 24 GB card alongside the weights.

Order has to be injected. Remove the positional encoding and “the dog bit the man” and “the man bit the dog” produce exactly the same output. This is not an implementation detail: it is a property of the attention operation, and every positional scheme is a correction bolted onto it. The practical consequence shows up in window extension: a model trained at 8 thousand tokens never saw the position angles corresponding to 100 thousand, and serving it a context far longer than its training degrades in ways short tests do not catch. Advertised window and usable window are different numbers.

Attention weights are not explanation. It is tempting to look at the attention map and conclude where the answer came from. A 2019 study found attention weights correlate poorly with gradient-based importance measures, and constructed very different attention distributions that produce the same prediction¹⁰. They are for inspecting, not for justifying.

Killing the quadratic term has not worked yet. This is worth recording because it is the busiest research axis around the block. One line attacks the cost without touching the mathematics: a 2022 paper reorganised attention to respect the GPU’s memory hierarchy and got a 3-times speedup on GPT-2 at thousand-token sequences while computing exactly the same thing¹¹. That improved the constant, not the exponent. The other line replaces the operation: state space models scale linearly with length, and in 2023 one of them matched transformers of twice its size at language modelling with 5 times the inference throughput¹². As of July 2026, frontier architectures remain predominantly full attention, sometimes interleaved with cheaper layers. The quadratic term is still there, just faster.

What to do with this

1. Treat context length as cost, not capacity. Every extra token pays twice: in attention’s quadratic term and in inference’s linear cache.
2. When sizing hardware, compute the cache before the weights. An 8-billion model fits in 16 GB, but 32 thousand tokens of conversation add several gigabytes nobody counted.
3. Be suspicious of comparing models by layer count. Depth and width trade off against each other, and the count alone says nothing about capability.
4. Do not use the attention map as justification in a report. For inspecting your own code it is fine; for explaining a decision to someone else it is not.
5. When reading a new architecture paper, find which of the four parts it swaps. Nearly all recent work touches position, normalisation, activation or head sharing, inside the same 2017 block.

Footnotes

1. Vaswani et al. (2017) describe each layer as two sublayers — multi-head attention and a position-wise feed-forward network — with a residual connection and normalisation around each. The base model used 6 layers per side, dimension 512, 8 heads of 64 dimensions and a feed-forward inner dimension of 2,048. ↩ ↩² ↩³ ↩⁴

2. Geva et al. (2020) analysed feed-forward layers as key-value memories: each key correlates with textual patterns and the matching value induces a distribution over the output vocabulary. ↩

3. He et al. (2015) introduced residual learning to make very deep networks trainable, reformulating layers as learning a residual with respect to the input. ↩

4. Xiong et al. (2020) showed, using mean field theory, that with normalisation after the residual block the gradients near the output are large at initialisation, which demands learning-rate warm-up. With normalisation inside the block, the warm-up can be removed. ↩ ↩²

5. Bahdanau et al. (2014) added to a recurrent translator a mechanism that searches the relevant parts of the source sentence for each generated word, removing the need to compress the whole sentence into a fixed vector. ↩

6. The Llama 3 report (2024) records 32, 80 and 126 layers for the 8, 70 and 405 billion parameter versions, with model dimension 4,096 in the smallest and 8 key-value heads against 32 query heads. ↩ ↩² ↩³

7. Tenney et al. (2019) found the steps of the classical language processing pipeline represented in order across BERT’s layers, with the model revising low-level decisions from higher-level information. ↩

8. Geva et al. (2022) read each feed-forward layer’s update in vocabulary space and used it to cut GPT-2’s toxicity by almost half and save 20% of the computation with an early exit rule. ↩ ↩²

9. Su et al. (2021) proposed encoding position by rotating the query and key vectors by an angle proportional to absolute position, which makes the relative dependency appear inside attention’s inner product. ↩

10. Jain and Wallace (2019) found weak correlation between attention weights and gradient-based importance measures, and constructed very different attention distributions that lead to the same prediction. ↩

11. Dao et al. (2022) made attention aware of reads and writes across GPU memory levels while keeping the computation exact: 3 times faster on GPT-2 at thousand-token sequences and 15% end to end on BERT-large. ↩

12. Gu and Dao (2023) integrated selective state spaces into an architecture with no attention at all, scaling linearly in length with 5 times the inference throughput. Their 3-billion model matched transformers of twice the size at language modelling. ↩