How is a language model trained?

An LLM is trained in phases. Pretraining runs once over trillions of tokens and installs the language and the facts. Supervised fine-tuning installs the shape of an answer. Alignment installs which answer people prefer. The first phase costs millions of dollars, and the other two together cost under 2% of that.

That split explains most of what people find frustrating about a foundation model in practice. Stale knowledge is a phase-1 problem and no amount of phase 2 will fix it. Trigger-happy refusals are a phase-3 problem and have nothing to do with how much the model knows. Unstable output formatting is a phase-2 problem, and it is the only one of the three an ordinary team can redo on its own.

The first two phases are conventional supervised training with different jobs: supervised fine-tuning over human-written demonstrations, and pretraining over raw text. The third, alignment from human feedback, is what turned a text continuer into something you could ship. They are worth taking one at a time.

The three phases

The order matters and does not reverse. You cannot align before there is something to align, and teaching output format to a model that cannot yet conjugate a verb buys nothing. Each phase starts from the weights the previous one produced.

Phase 1: pretraining, where the knowledge enters

The objective is the simplest of the three: given a stretch of text, predict the next token. No human labels, no defined task. The text is its own answer, which is what makes it possible to consume volumes no manual annotation could ever reach.

The corpus is the most consequential and least documented decision in the industry. The public example you can actually inspect is the Pile, from 2020: 825 GiB of English text assembled from 22 distinct subsets, many of them academic or professional in origin. The authors themselves documented concerning aspects of the data for anyone planning to use it¹. Frontier corpora in 2026 are an order of magnitude larger and none of them is published.

Scale became predictable in 2020 and proportional in 2022. GPT-3 used 175 billion parameters over 300 billion tokens², under two tokens per parameter. A 2022 study trained more than 400 models to find the optimal allocation and showed the models of that era were undertrained: every doubling of parameters should come with a doubling of tokens. The model they used to prove it, 70 billion parameters over 1.4 trillion tokens, beat the 280-billion Gopher on the same compute budget³. Four years later, DeepSeek-V3 was pretrained over 14.8 trillion tokens⁴, close to 50 times the GPT-3 corpus.

What comes out of this is a base model. It knows a great deal and obeys nothing.

What a run like this actually looks like

The operation is less glamorous than the result suggests. Thousands of accelerators chew through batches of text in parallel, each computing its prediction error on its own slice; the gradients are summed across every machine and applied to the weights. That cycle repeats hundreds of thousands of times across weeks or months of wall-clock time, with no planned interruption.

What dominates the engineering is failure. A GPU dies mid-run and the whole job stops if there is no recent checkpoint. The loss spikes for no visible reason and training has to roll back to an earlier state. The DeepSeek-V3 report makes a point of recording that the entire run had no irrecoverable loss spikes and no rollbacks⁴, which is only worth mentioning because the opposite is normal. A frontier pretraining run is, in large part, an exercise in distributed fault tolerance with a neural network inside it.

This is also why you cannot “pick up where it left off” with fresh data and call the model current. Injecting a recent corpus into a trained model is continued pretraining, with cost proportional to the new volume and a real risk of degrading what was already there. Nobody does that weekly, and that is why the knowledge cut-off exists.

Phase 2: supervised fine-tuning, where the format enters

Here somebody writes the answers. The InstructGPT set, in 2022, held around 13 thousand training prompts, part drawn from the API and part written by the annotators themselves — a team of 40 people hired after a screening test⁵. Each example is a pair: the request, and the response considered good.

Two things surprise people who arrive expecting scale.

The first is that the volume is tiny. Thirteen thousand examples, against 300 billion tokens in the previous phase. The second is that cutting it further seems to work. A 2023 study tuned a 65-billion-parameter model on a thousand hand-curated prompts and responses, with no reinforcement learning at all, and got answers human raters judged equal to or better than GPT-4’s in 43% of cases. The authors’ conclusion is the interesting part: almost all the knowledge is already in the model after pretraining, and tuning serves to teach which response format to use⁶.

That changes what you optimise for. In pretraining, more data is almost always better. In supervised fine-tuning, consistency beats volume: a thousand examples that agree with each other about what an answer should look like produce a better model than a hundred thousand that disagree.

Phase 3: alignment, where the preference enters

The third phase does not ask anyone to write the right answer. It asks someone to pick between two. That matters because comparing is far cheaper and far more reliable than drafting: two people disagree a lot about how to write the ideal answer and agree considerably more about which of two answers is better.

The classic recipe has three steps. Annotators rank the model’s own outputs — the InstructGPT reward set held 33 thousand training prompts. A reward model learns to predict those rankings. And the LLM is optimised against that reward model by reinforcement learning, over a third set of 31 thousand prompts carrying no human labels at all⁵.

Two simplifications reshaped this stage afterwards.

A 2023 paper showed the separate reward model is dispensable: you can derive a simple classification loss over the preference pairs and optimise the LLM directly, with no sampling during training and none of the reinforcement learning hyperparameters⁷. That dropped the barrier to entry considerably, and it is why small teams can now run their own alignment.

The other route attacks the cost on the human side. A 2022 approach replaced most harmlessness labels with feedback from the model itself, judged against a written set of principles⁸. Human labelling does not disappear, but it gets spent where no model substitutes for it.

The same prompt after each phase

The first column is the confusing one. The base model does not get photosynthesis wrong; it never attempts it. It recognises that “Explain photosynthesis to a child” looks like an item in a list of exercises, and writes the next item. It is doing exactly the task it was trained on.

The second column answers, and answers correctly. What it ignores is the “to a child” part, because supervised tuning taught the shape of an explanation rather than sensitivity to the whole request. The third column is the one that registers that the reader is six. That gap between the second and the third is easy to underrate, and it is much of what separates a usable model from one people actually enjoy using.

The fourth phase, which showed up later

Since 2024, reasoning models have added a reinforcement stage over tasks with verifiable answers, where the reward comes from a checker rather than a human preference: the test passed, the arithmetic checks out, the proof closes.

The result that made this public came from DeepSeek in 2025. They showed reasoning ability can be incentivised through pure reinforcement learning, with no human-labelled reasoning trajectories, and that patterns like self-verification and mid-course strategy changes emerge on their own during that training⁹. The resulting model beat counterparts trained by supervision over human demonstrations on mathematics, coding competitions and STEM tasks.

The caveat holds: this works wherever an automatic checker exists. For “write a good apology email” there is none, and phase 3 remains the only answer.

What each phase costs

The InstructGPT numbers are the most useful ones because all three phases appear in the same unit. Training the 175-billion-parameter model with supervised fine-tuning consumed 4.9 petaflop/s-days. The reinforcement version consumed 60. The GPT-3 pretraining run underneath both consumed 3,640⁵. Added together, the entire post-training came in under 2% of what the previous phase cost.

The most transparent report on the money side is DeepSeek-V3’s. The paper’s cost table separates 2.664 million H800 GPU-hours for pretraining, 119 thousand for context extension and 5 thousand for post-training, totalling 2.788 million hours. At the two-dollar-per-GPU-hour rental price they assume, that is 5.328 million dollars, 238 thousand and 10 thousand, for a total of 5.576 million⁴. The authors record that this figure covers the official run and excludes prior research, ablations and discarded experiments. A realistic estimate of a lab’s total spend is several times that.

The ratio is the part that ages well. Absolute figures shift with every hardware generation, but the shape of the bill has held for six years: the phase that installs knowledge dominates the budget, and the phases that install behaviour decide whether the model is good for anything.

Where it breaks

The corpus contains the benchmark. When the training text comes from the internet and the test does too, it is hard to guarantee that one does not contain the other. This is the central problem of LLM evaluation, and it is born here, in phase 1, not at measurement time.

Alignment charges a capability tax. Preference optimisation tends to degrade performance on academic tasks relative to the supervised-only model. InstructGPT handled it by mixing pretraining gradients into the reinforcement stage, which is where the “ptx” suffix in their model name comes from⁵. Teams running their own alignment without that precaution usually find the regression later.

The model learns to please the rater, not to be right. The reward is a model trained on human judgements, and the LLM is optimised against that model. Anything that fools the reward model gets rewarded: longer answers, more confident ones, better formatted ones. A correct answer and a convincing answer are not the same object, and the gradient cannot tell them apart. The clearest everyday symptom is the reply that opens by restating your question, adds three bullet points of caveats and closes by offering to go deeper. None of that was asked for. It survives because a rater with two answers and little time reliably picks the one that looks more thorough.

Annotators are a specific population. Forty people shaped much of InstructGPT’s behaviour⁵. Nothing is wrong with that, but it is worth knowing that “what people prefer” in phase 3 means what that group preferred, following those instructions, at that moment.

New knowledge does not enter through post-training. That follows directly from the 2023 thousand-example result⁶. If the fact was not in the corpus, you have to put it into the context at question time, not into the tuning set.

What this changes for people who only use the model

1. When you see odd behaviour, name the phase. A wrong or stale fact is phase 1: put the information in the context. Unstable formatting is phase 2: give examples in the prompt. A refusal or a misplaced tone is phase 3: work on the system instructions, not the content.
2. Do not try to teach knowledge by fine-tuning. It is the most expensive route to the worst outcome. Tuning teaches behaviour; knowledge goes in the context.
3. If you do tune, spend on the consistency of the set, not its size. A thousand examples reviewed by one person beat ten thousand written by five people working from different standards.
4. Pin the exact model version. A provider can ship new post-training under the same commercial name, and phase 3 is what moves first.
5. Test refusals and tone separately from capability. Different phases control them, and they move independently between versions.

Footnotes

1. Gao et al. (2020) assembled an 825 GiB corpus from 22 distinct subsets and documented concerning aspects of the data for prospective users. ↩

2. Brown et al. (2020) trained eight models up to 175 billion parameters, all over 300 billion tokens, and showed the task can be specified by examples inside the input text itself. ↩

3. Hoffmann et al. (2022) trained more than 400 models and concluded the models of that era were undertrained: parameters and tokens should scale together. Their 70B model over 1.4T tokens beat the 280B Gopher on equal compute. ↩

4. Table 1 of the DeepSeek-V3 report (2024) separates 2.664 million H800 GPU-hours for pretraining, 119 thousand for context extension and 5 thousand for post-training, at two dollars per GPU-hour. ↩ ↩² ↩³

5. Ouyang et al. (2022) describe the three sets used: 13 thousand supervised fine-tuning prompts, 33 thousand reward prompts and 31 thousand reinforcement prompts, labelled by a team of 40 contractors. Reported compute was 4.9 petaflop/s-days for the 175B supervised model and 60 for the reinforcement version, against 3,640 for GPT-3 pretraining. ↩ ↩² ↩³ ↩⁴ ↩⁵

6. Zhou et al. (2023) tuned a 65B model on a thousand curated prompts and responses, with no reinforcement learning, and got answers rated equal to or better than GPT-4’s in 43% of the cases evaluated. ↩ ↩²

7. Rafailov et al. (2023) showed the explicit reward model is dispensable: the optimal policy has a closed form that allows optimising directly over preference pairs with a classification loss. ↩

8. Bai et al. (2022) trained a harmless assistant by replacing most human harmlessness labels with feedback from the model itself, judged against a written set of principles. ↩

9. DeepSeek (2025) showed reasoning ability can be incentivised through pure reinforcement learning, with no human-labelled trajectories, with self-verification and strategy adaptation emerging during training. ↩