What is a large language model (LLM)?

A large language model (LLM) is a neural network trained to predict the next token in a piece of text. Nearly all of them are built on the transformer architecture, go through two training phases, and come out general enough to serve as a foundation model: they handle tasks nobody programmed into them.

The “large” describes the model, not the text it reads. Billions of parameters are fitted over trillions of tokens, and everything else follows from that quantity. Nothing in training defines what a question, a translation or a code snippet is. Those categories show up because predicting the next word well across a big enough corpus turns out to require recognising all of them.

Next-token prediction is the whole job

There is no second, hidden operation. There is a four-step cycle, run once per generated token.

Tokenize. The text is broken into subword pieces and each piece becomes a number. “Roof” may be one token or three, depending on how often that sequence appeared in the corpus the tokenizer was fitted on. That detail explains a good share of the silly failures further down.

Attention. Each token works out how much every other token in the input matters to it, and builds its representation from that. This is what replaced the recurrence of earlier architectures: instead of walking the sentence word by word, the model compares all positions at once¹.

Score the vocabulary. The last layer produces a number for every possible token, somewhere between 30,000 and 200,000 options depending on the tokenizer. A normalisation step turns those numbers into probabilities.

Pick. The system draws a token from that distribution. At temperature zero, always the most likely one; above zero, in proportion to probability. This is the one place randomness enters.

The chosen token is appended to the input and the cycle starts over. A 500-token answer is the whole model running 500 times, rereading everything before it each time. That is why output costs more than input, and why answers arrive on screen word by word.

What attention solved

The paper that introduced the transformer, in 2017, was not about chatbots. It was about machine translation, and the headline result was 28.4 BLEU on WMT 2014 English-to-German¹. The score is not what stuck. What stuck is that without recurrence, any position in the sequence reaches any other in a single step, and training parallelises. The authors’ big model took 3.5 days on eight GPUs, a fraction of what the architectures it beat had cost.

There is a price baked into that. Attention compares every token with every other one, so cost per layer grows with the square of sequence length. Double the context and you quadruple that work. Much of the long-context engineering of the following years is an attempt to work around that quadratic term.

What scale bought

In 2020, two results changed how model size gets decided. The first showed that training loss falls as a power law in parameter count, dataset size and compute, with the trend holding across more than seven orders of magnitude². That made scale predictable: you can estimate the gain before spending the money.

The second fixed the ratio. GPT-3 had 175 billion parameters and was trained on 300 billion tokens³, under two tokens per parameter. A 2022 paper trained more than 400 models to find the optimal allocation and concluded that size and data should grow together: every doubling of parameters calls for a doubling of tokens. The model they trained to prove it, at 70 billion parameters and 1.4 trillion tokens, spent the same compute as the 280-billion Gopher and beat it, along with models of 175, 178 and 530 billion⁴.

Twenty tokens per parameter became the rule of thumb from there. In practice that retired “how many billions” as a standalone question, and moved the contest to data and to what happens after pre-training.

A second caveat took apart what was left of that reading. In a mixture-of-experts model, each token crosses only a slice of the network: total parameters grow without cost per token following, because most of them sit idle on any given pass⁵. The technique dates from 2021 and became common among large models, so comparing two models by total parameter count can compare different things. One of them may be activating a tenth of what it advertises.

There is a third effect, more debated than settled. A 2022 paper catalogued abilities absent in small models that appear abruptly above a certain scale, and called them emergent⁶. A 2023 paper argued that much of that sharpness is an artefact of the metric: with discontinuous metrics, such as requiring an exact match, the curve looks like a step; with continuous metrics over the same model outputs, it smooths back out⁷. The fair reading is that capability grows more continuously than the 2022 charts suggested, and that “it only works above X billion” deserves suspicion.

What you download when you download a model

An open model is a file of numbers plus a little architecture metadata. Weights are usually published at 16 bits, which is two bytes per parameter: an 8-billion model takes about 16 GB, a 70-billion one about 140 GB. That figure is the first filter for running locally, because it has to fit in accelerator memory alongside the attention cache of the conversation in flight.

Quantization drops each weight to 8 or 4 bits and shrinks the file in the same proportion: that 70-billion model lands near 35 GB at 4 bits. Quality drops too, by an amount that varies with the task and that only your own set of cases can measure. A closed model served over an API has neither that problem nor that freedom: you never see the weights, and the version can change under you.

Two training phases, and the second one sets behaviour

Pre-training produces a model that continues text. It does not answer questions. It predicts what would follow a question in an internet document, which might be another question, a list of similar questions or a forum footer.

Post-training turns that into something that answers you. It runs in two stages: supervised tuning on human-written examples of the desired answer, then optimisation against human comparisons between the model’s own outputs. A 2022 paper measured the effect in a way that is hard to argue with: raters preferred the answers of a 1.3-billion-parameter model treated this way over those of a 175-billion-parameter model without it⁸.

It pays to keep straight what each phase delivers. Knowledge comes from pre-training; post-training adds almost no new facts. Format, refusal, tone and willingness to follow instructions come from post-training. When two models seem to know the same things but one “is nicer to use”, the difference usually lives there rather than in size.

Since 2024 that phase has grown a third stage: training the model to spend tokens working the problem before answering, using reinforcement learning on tasks with checkable answers. That is what separates a reasoning model from an ordinary one, and the difference shows up on the bill, because some of those tokens are charged without ever being shown. As of July 2026 it looks more like a mode you switch on than a separate family of models.

The context window is not memory

The context window is the token ceiling of a single call, covering what goes in and what comes out. GPT-3 worked with 2,048 tokens in 2020³; as of July 2026, frontier models advertise windows between a few hundred thousand tokens and a million. The gap is real and it changed what you can build.

What did not change is the nature of the thing. There is no state between calls: a twenty-turn conversation works because the client resends all twenty turns on every request. The model does not remember the previous message, it rereads it. That is why a long conversation costs more each turn even when you type very little.

A large window does not mean uniform use of it either. A 2023 paper measured performance while moving the relevant information around inside the context and found a U-shaped curve: accuracy is high when the useful passage sits at the start or the end, and drops sharply when it sits in the middle, including in models sold as long-context⁹. The practical reading is blunt: put important instructions at the start or the end, and do not bury the document you actually need read in the middle of ten others.

The five axes of the subject

The rest of the topic organises into five axes, and naming the axis a question belongs to already rules out half the wrong answers to it.

Architecture is the part that barely varies: the transformer with attention, in variants that move efficiency more than capability. Training is where the expensive decisions sit. Families is the commercial axis, and the only one that compares Claude, GPT, Gemini and Llama without turning into a brochure. Evaluation is what benchmarks measure and what they miss. Operation is window, cost, latency and the rest of what decides whether the thing ships.

Where it fails

It does not know what it does not know. The training objective rewards plausible continuation. No step checks whether the output is true, so an invented citation is produced by exactly the same machinery as a correct one. The more specific and less frequent the fact, the higher the risk. The easiest case to reproduce is asking for five academic papers on a narrow topic: they come back with plausible authors, a plausible journal and a correctly formatted DOI, and some of them do not exist. The shape is right because shape is what the model learned; existing is not a property the training objective checks.

Tokenization leaks into character-level tasks. Counting letters, reversing a word or reasoning about spelling fails more than the model’s apparent competence suggests, because it does not see letters. It sees tokens that bundle several of them together.

Multi-digit arithmetic is brittle. The model learned the patterns of arithmetic, not the algorithm. Multiplying large numbers is where this shows up first, and the fix is to hand the calculation to a calculator or to code rather than push harder on the prompt.

Knowledge has a date. What was not in the pre-training corpus is not in there, and the model has no reliable sense of where its coverage ends. For facts that move, the answer is to fetch the data and put it in the context.

The same question can give different answers. Above temperature zero, token choice is a draw. Even at temperature zero, request batching and numerical differences in hardware mean exact repeatability is not something you get in practice.

Where to start

1. Write twenty real cases before picking any model. Real inputs, real expected outputs. It is the only instrument that answers your specific question.
2. Start with the small model in the family. If it passes the twenty cases, the big one is money and latency thrown away.
3. Measure three things separately: accuracy, time to first token and cost per call. They move in different directions, and the decision is almost always about which pair you accept degrading.
4. Pin the model version in production. Pointing at the floating alias means your application changes behaviour without you shipping anything.
5. Only then touch the prompt. Most of the cheap gain lives in prompt engineering, not in switching models.

Cost, as an order of magnitude

What gets billed is tokens, split into input and output. As of July 2026, output costs several times more than input at the large providers, which makes a summarising task far cheaper than a generating one even when both read the same document.

Two estimates help you size this before opening a price list. English text runs close to 1.3 tokens per word, so a 2,000-word article goes in at roughly 2,600 tokens. And the stable part of a request, such as the system prompt and any fixed examples, can be served from cache by providers that offer prompt caching, at a steep discount on the repeated stretch. If your application sends the same 4,000-token block on every call, that is the first optimisation to make, ahead of switching models.

Footnotes

1. Vaswani et al. (2017) proposed an architecture based on attention alone, with no recurrence or convolution, and reported 28.4 BLEU on WMT 2014 English-to-German. Table 1 of the paper records the trade: O(n²·d) complexity per self-attention layer, against a constant-length path between any two positions. ↩ ↩²

2. Kaplan et al. (2020) measured loss against size, data and compute, and found stable power laws spanning more than seven orders of magnitude, with width and depth mattering little inside a broad range. ↩

3. Brown et al. (2020) trained eight models up to 175 billion parameters, all for 300 billion tokens and with a 2,048-token context window, and showed the task can be specified by examples in the text itself, with no weight updates. ↩ ↩²

4. Hoffmann et al. (2022) trained over 400 models, from 70 million to more than 16 billion parameters, and concluded that the large models of the day were undertrained. Chinchilla, at 70B and 1.4T tokens, beat Gopher (280B), GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B). ↩

5. Fedus et al. (2021) simplified mixture-of-experts routing and trained sparse models with very large parameter counts at constant computational cost per token, reporting up to 7x faster pre-training than the comparable dense baseline. ↩

6. Wei et al. (2022) defined an ability as emergent when it is absent in smaller models and present in larger ones, and argued it cannot be predicted by extrapolation. ↩

7. Schaeffer et al. (2023) showed that discontinuous metrics manufacture apparent jumps over unchanged model outputs, and reproduced the effect deliberately on vision tasks. ↩

8. Ouyang et al. (2022) tuned GPT-3 on human demonstrations and then with reinforcement learning over human rankings. Outputs from the 1.3B model treated this way were preferred to those of the 175B GPT-3. ↩

9. Liu et al. (2023) varied the position of the relevant document in multi-document question answering and key-value retrieval, and found significant degradation when the information sits mid-context. ↩