What changes from one model generation to the next?

A model generation is a jump that changes what you can build, not just how often the model gets things right. Five axes move between one generation and the next: scale, data, alignment, context window, and the compute spent at answer time. No release moves all five. Working out which one moved is what tells you whether the news matters for your workload.

There is no official definition of “generation”. Whoever ships the model picks the version number, and that number answers to a release calendar as much as to engineering. The test that works is different: if your application has to be rewritten to take advantage of the new model, that was a generation; if it just gets fewer things wrong on the same cases, that was a point release. The machine underneath all of them is the same one, described in what an LLM is.

This page covers the mechanism: the axes, and the bottleneck pattern that repeats at every jump. The sibling articles cover the episodes — what GPT-2 could already do, the reasoning model era, the state of frontier models right now and the LLM timeline.

The five axes

Every release announces itself as “more capable”, which is not information. The information is in which axis moved, because each one buys something different and each one bills differently.

That split is where the confusing part of the bill comes from: a generation can get cheaper and more expensive at the same time, with a lower price per token and more tokens per task.

Scale

This was the dominant axis from 2018 to 2022. A 2020 paper showed that training loss falls as a power law in parameter count, data volume and compute, with the trend holding across more than seven orders of magnitude¹. Scale became a predictable investment: you could estimate the gain before spending.

The axis still moves, it just stopped being announced. The GPT-4 technical report, in 2023, states outright that it contains no details about architecture, model size, hardware, training compute or dataset construction, and gives the competitive landscape as the reason². Anyone comparing generations by parameter count had that instrument taken away that year and has not got it back.

Data

In 2022, a paper trained more than 400 models to find the optimal split between size and data and concluded that the models of the day were undertrained: every time you double parameters, double the tokens. Their 70-billion model trained on 1.4 trillion tokens beat Gopher, at 280 billion, on the same compute budget³. The question stopped being “how many billions” and became “how many tokens, and which ones”.

The “which ones” part is what nobody publishes. Share of code, language mix, how much of the corpus is synthetic, what got filtered out — none of it shows up in the announcement, and it is where the large differences between similarly sized models live. When a new generation is visibly better at code and at nothing else, the data mix is what changed.

Alignment

This is the axis with the best ratio of perceived gain to cost. A 2022 paper fine-tuned GPT-3 on human-written demonstrations and then with reinforcement learning over human comparisons between the model’s own outputs. Raters preferred the answers of the 1.3-billion-parameter model treated that way to those of the 175-billion model without the treatment⁴. Roughly 135 times fewer parameters, and still preferred.

That result explains a lot of the confusion around generations: a model can get far more useful without getting bigger and without seeing more data. It also runs the other way. A new generation can follow the provider’s policy more closely and refuse more often, which reads as progress to whoever ships it and as a regression to whoever had an application working at the edge of that policy.

Context window

Here the number is always announced, always checkable, and almost always over-read. GPT-3 worked with 2,048 tokens in 2020⁵. In 2024, the Gemini 1.5 report logged above 99% retrieval on long-context tasks out to at least 10 million tokens in experiment, and compared that against what existed at the time: 200 thousand tokens on Claude 3.0 and 128 thousand on GPT-4 Turbo⁶. The scale of this axis is real and it changed what can be built. Pasting a whole repository into the prompt stopped being a joke.

What does not follow the number is uniform reading across the window. An advertised window is an input ceiling, not a promise of attention, and two generations with the same nominal ceiling behave quite differently at 300 thousand tokens. That distance is what the context window article is about.

Compute at answer time

This is the newest axis and the one that most disturbed the bill. Instead of spending more in training, you spend more when answering: the model writes tokens working the problem before it replies, or the system generates several attempts and picks among them.

A 2024 paper studied how to allocate that budget and found the effective strategy depends on how hard the prompt is; allocating adaptively, the authors report the same quality for far less test-time compute, and regimes where this pays better than having scaled pre-training parameters⁷. Another paper from the same year measured the blunt version, sampling repeatedly: coverage, the fraction of problems solved by any of the samples, grows log-linearly across four orders of magnitude. On SWE-bench Lite, one model went from 15.9% with a single sample to 56% with 250⁸.

The bottleneck of the axis is inside that same result. It works because code has an automatic verifier: you run the tests and find out which sample is any good. Without a verifier, the usual ways of picking among samples — majority voting and reward models — plateau after a few hundred and stop tracking the budget⁸. Inference compute buys accuracy wherever something can say whether an answer is correct. Where nothing can, it buys tokens.

The map of the subject

The eras are the jumps and what each one unlocked. This is the part that stops changing once it has happened, and the safest place to start.

The families are the commercial axis: GPT, Claude, Gemini, Llama, Qwen, DeepSeek. Each has its own cadence, its own naming convention and its own criteria for what counts as a major version. That is also where the open-weights escalation lives.

The current state is the perishable branch: who is ahead, what it costs, what window it offers.

Reading a release is the operational part: model cards, changelogs, snapshot or alias, deprecation timelines and why the names are so confusing.

The eras, and the bottleneck each one opened

2020, few-shot. GPT-3 showed that the task can be specified by examples inside the text itself, with no weight updates at all⁵. Programming by prose became possible. The bottleneck it opened was immediate: the model still continued text, it did not obey. Getting a usable answer meant building the prompt the way you build a trap.

2022, instruction. Post-training on human feedback fixed exactly that bottleneck⁴, and the product built on it carried the whole thing outside the technical community. The new bottleneck was measurement: once usefulness became the thing being judged, benchmark scores stopped standing in for “this is good to work with”, and the yardstick moved towards human preference comparisons.

2023 and 2024, multimodal and long context. GPT-4 took images alongside text² and windows went from thousands to millions⁶. Whole documents and screenshots became ordinary input. The bottleneck: cost and latency per call grew with them, and the advertised window settled above the window the model actually uses well.

2024 to 2026, reasoning and action. Spending tokens before answering became a training technique, not just a prompting one. A 2025 paper showed that pure reinforcement learning, with no human-annotated reasoning traces, is enough for the model to develop self-verification and mid-course strategy changes⁹. That, plus training to call tools in a loop, took on multi-step tasks. The bottleneck is still open: cost per task rather than per token, latency the user feels, and how to grade a twenty-step path with an instrument that measures single answers.

The pattern is worth more than the list. Anchoring your architecture on what the current generation does well ages badly; anchoring on the bottleneck it opened ages well, because the open bottleneck is the agenda for the next eighteen months.

How to read a release

The idea of documenting a shipped model dates to 2019: a short card with intended use, evaluation procedure and performance broken down by subgroup, so the model would not end up running where nobody tested it¹⁰. What frontier labs publish today is a blend of spec sheet, safety report and scoreboard, with the technical half retreating every generation².

In practice, four questions pull out most of what a release announcement holds:

1. Which axis moved? Gains concentrated in maths and code, with slower answers, are inference compute. Gains in instruction following and fewer mistaken refusals are post-training. A bigger window always ships with a number. Scale and data you infer from everything else.
2. Compared against what, and when? A score is always against something on some date, and a comparison against the house’s own previous generation says less than it looks like it does.
3. Snapshot or alias? A dated name points at fixed weights. A generic name points at whatever the provider decides to serve tomorrow.
4. What was retired? Every release pushes an older model into the deprecation queue, and the date is usually in the footer.

Where it breaks

“Generation” is a marketing label. A version announced as minor sometimes carries a bigger change than the one before it. The label is not a measurement.

The same name can change behaviour with no notice. A 2023 paper compared the March and June versions of GPT-3.5 and GPT-4 across seven tasks. GPT-4 scored 84% classifying prime numbers in March and 51% in June; adherence to chain-of-thought prompting dropped; both models started making more formatting mistakes in generated code¹¹. Versioning practices improved after that, but the risk only disappears when you pin the version.

Benchmarks saturate before usage does. Once a score approaches the ceiling, the difference between two generations vanishes from the metric long before it vanishes from the real task, which is why what benchmarks measure matters more with every generation.

There is no guarantee of monotonic improvement on your workload. The lab measured what the lab has. A newer model can get worse on an input type that only exists in your corpus, and you only find out with your own suite.

Where to start

1. Keep twenty to thirty real cases with the expected output. It is the only instrument that answers your question instead of the lab’s. Without it, every migration is guesswork with a technical accent.
2. Before migrating, answer which axis moved. If it was inference compute and your task has no verifier, the likely gain is small and the bill goes up.
3. Pin the snapshot in production. A generic alias means your behaviour changes without you having deployed anything.
4. Run the old cases on the new model before touching the prompt. Swapping model and prompt together destroys the only comparison you had.
5. Measure cost per task, not per token. It is the only way to compare a generation that reasons against one that answers directly.
6. Keep the old model reachable through an environment variable. The regression that escapes your suite shows up on a Friday.

What switching generations costs

Orders of magnitude, to know the shape of the arithmetic before opening a price list.

Price per token falls with every generation, and that is the easy half to work out. The surprising half is token quantity: when the axis that moved was compute at answer time, the same question starts consuming tokens you pay for and never see, and comparing unit prices between two generations stops meaning much. The repeated-sampling experiment is a ruler for the far end of that curve: 250 samples cost 250 times one sample, and that was the price of the jump from 15.9% to 56% on SWE-bench Lite⁸. Worth it on a task expensive enough; not worth it in autocomplete.

The rest of the cost is not on the provider’s invoice. Migration charges you a prompt rewrite, a fresh evaluation round and adjustments to everything that depended on the exact previous output shape. On a small team that usually exceeds the token savings of an entire first quarter, which is a good argument for migrating on capability gains rather than on pennies.

Footnotes

1. Kaplan et al. (2020) measured loss as a function of size, data and compute and found power laws holding across more than seven orders of magnitude. ↩

2. The GPT-4 technical report (2023) describes a model that accepts image and text input, and states that it contains no details about architecture, size, hardware, training compute or dataset construction, citing the competitive landscape and safety implications. ↩ ↩² ↩³

3. Hoffmann et al. (2022) trained more than 400 models and concluded that the models of the time were undertrained. Chinchilla, at 70B with 1.4T tokens, beat Gopher (280B) and other larger models on the same compute. ↩

4. Ouyang et al. (2022) fine-tuned GPT-3 on human demonstrations and then with reinforcement learning over human rankings. Outputs from the 1.3B model treated that way were preferred to those of the 175B GPT-3. ↩ ↩²

5. Brown et al. (2020) trained eight models up to 175 billion parameters with a 2,048-token window and showed the task can be specified by examples in the text itself, with no weight updates. ↩ ↩²

6. The Gemini 1.5 report (2024) reports above 99% retrieval out to at least 10 million tokens of context, comparing against Claude 3.0 (200 thousand) and GPT-4 Turbo (128 thousand). ↩ ↩²

7. Snell et al. (2024) analysed search against process-based verifiers and adaptive updates to the response distribution: how well each method works varies with prompt difficulty, which is what motivates allocating test-time compute per prompt. ↩

8. Brown et al. (2024) showed coverage growing log-linearly with sample count across four orders of magnitude, with DeepSeek-Coder-V2-Instruct going from 15.9% to 56% on SWE-bench Lite at 250 samples. Without an automatic verifier, majority voting and reward models plateau after a few hundred. ↩ ↩² ↩³

9. DeepSeek-R1 (2025) showed that pure reinforcement learning, with no human-annotated reasoning trajectories, incentivises self-reflection, verification and strategy switching, with gains on verifiable maths and coding tasks. ↩

10. Mitchell et al. (2019) proposed short cards shipping alongside released models, covering intended use, evaluation procedure and performance by demographic subgroup. ↩

11. Chen et al. (2023) evaluated the March and June 2023 versions of GPT-3.5 and GPT-4 across seven tasks and found large behaviour swings, including GPT-4 dropping from 84% to 51% on prime number identification. ↩