What is a foundation model?

A foundation model is a model trained once over an enormous amount of general data and then adapted to many tasks nobody had in mind while it was training. The term describes where the model sits in the chain, not what it is built from: it is the base other models are built on, either by moving weights or by prompt alone.

For almost every text foundation model in use today, the answer to “what kind of model is that?” is an LLM. The two phrases cover nearly the same set of files but answer different questions. “LLM” says what the thing is: a network trained to predict the next token. “Foundation model” says what it is for: a reusable part, trained by whoever owns a cluster and used by everyone who does not.

The gap shows up in the bill. Pretraining costs millions of dollars in compute and takes months. Deriving a specialised model from a pretrained one costs somewhere between tens and a few thousand dollars, and takes hours. As long as that ratio sits at five or six orders of magnitude, the division of labour holds: few organisations train foundations, many derive from them. That is why nearly every AI model in production today is an adaptation of something another company trained.

Where the term came from

The name landed in 2021, in a report from the Stanford centre set up to study these systems. The authors looked at BERT, DALL-E and GPT-3 and noticed the three shared something existing categories missed: they were trained on broad data at scale and adaptable to a wide range of downstream tasks. They picked “foundation” to mark that central and simultaneously incomplete character — the model is the base of something else, and on its own it is not a product¹.

The report’s most useful argument is not the definition. It is the consequence they called homogenisation: when thousands of applications sit on the same handful of bases, the defects of the base are inherited by everything built on top. A bias in the pretraining corpus, a language gap, a knowledge cut-off, a tokenisation flaw. None of that gets corrected three layers up, and much of it is not even visible from there.

Homogenisation is why “which model is under this?” became an architecture question rather than idle curiosity. Two applications that have never heard of each other can fail the same way on the same day because they share a foundation.

Adapting means three different things

The cheapest adaptation touches no weights at all. The GPT-3 paper, in 2020, showed you can specify the task inside the input text itself, with a few examples, and no gradient update anywhere². That is what we now call prompting, and it is how most applications adapt a foundation. It costs one API call and reverts instantly.

The middle route moves a small slice of the weights. A 2021 paper proposed freezing the whole model and training only low-rank matrices injected into each layer; on GPT-3 at 175 billion parameters this cut trainable parameters by 10,000 times and GPU memory by 3 times, matched full fine-tuning on quality, and added no inference latency³. That result is what made it practical for an ordinary company to keep dozens of variants of one model around. Each variant is a file of a few megabytes sitting beside the original weights, not a copy of the model. Supervised fine-tuning is the usual way to train that slice.

The expensive route rewrites every weight. That is continued pretraining or full fine-tuning, and the cost climbs back toward the original training run. Outside a lab it is almost never the right call: it is expensive, it needs a lot of data, and it tends to damage what the model already knew.

What separates a base model from an instruct model

A base model fresh out of pretraining does not answer questions. It continues text, which is the only task it was ever trained to do. Type “How do I reboot the router?” and it may hand you three more questions like it, because in a great many web documents that is exactly what follows a question: a list of frequently asked ones.

Assistant behaviour gets installed afterwards, in two stages. First supervised tuning on examples written by people showing the desired answer. Then optimisation against human comparisons between the model’s own outputs. The 2022 paper that documented this recipe measured the effect in a way that is hard to argue with: human raters preferred the answers of a 1.3-billion-parameter model treated this way over those of a 175-billion-parameter model without it⁴.

In practice you find both files published side by side. Meta shipped Llama 2 exactly that way, a collection of pretrained models from 7 to 70 billion parameters together with the chat versions derived from them⁵. The repository name is usually what tells them apart: -base or no suffix for the pretrained weights, -instruct, -chat or -it for the derivative.

Picking the wrong one is cheap to do and expensive to diagnose. A base model behind a chat endpoint produces output that looks like a prompt bug: it ignores the instruction, echoes the question, loops. It is not a bug. It is the model doing its job.

What the base decides and tuning cannot undo

It is worth being specific about what gets frozen the moment a foundation finishes training.

The knowledge. The most direct evidence came from a 2023 study that tuned a 65-billion-parameter model on a thousand carefully curated examples, with no reinforcement learning at all, and got answers human raters judged equal to or better than GPT-4’s in 43% of cases. The authors concluded that almost all knowledge is learned during pretraining, and that tuning mostly teaches which response format to use⁶. If the fact was not in the corpus, a thousand tuning examples will not put it there.

The tokeniser. It is fixed before pretraining and never changes afterwards. A tokeniser that shreds your language into many pieces makes every call in that language more expensive, permanently, across every model derived from that base. The cost is not only money: more tokens per word also means the same document eats more of the context window, so the effective window of a badly tokenised language is smaller than the number on the spec sheet. You inherit all of that without ever having been in the room for the decision, and you can measure it in ten minutes by running one real paragraph through two families’ tokenisers and comparing the counts.

The context window. Extending context after pretraining is possible, but it is a training procedure in itself, not a setting. Whatever ceiling the base supports is the practical ceiling of its derivatives.

Gaps in language and domain. If the corpus held little legal Brazilian Portuguese, tuning will teach the model to sound like a Brazilian lawyer without teaching it what the law says. That combination is worse than obvious incompetence, because the confident tone survives the missing substance.

The category is not only about text

The 2021 report already handled vision, robotics and human interaction in the same breath as language¹, and the reason is that the recipe does not depend on the modality: train broad once, adapt many times. The pattern now shows up in images, audio, code, time series and protein structure, and the same cost asymmetry repeats in each of them.

What changes across modalities is which adaptation route actually works. In text, prompting handles most cases, because the instruction and the data are the same substance: everything is a token. Outside text that route is weaker. An image foundation model rarely accepts natural-language instruction with the same precision, and adaptation tends to fall back to training, even if only low-rank. People carrying LLM intuitions into another modality tend to discover this late, after writing prompts for a model that was never built to receive them.

Multimodal models blur the line, and it is worth being precise about what they are. A model that takes both image and text as input is not two foundation models glued together: it is one foundation, trained over pairs of both kinds of data. The ability to look at a figure and answer questions about it comes out of the joint pretraining, not from a vision component bolted on afterwards.

Where it breaks

Tuning for your domain degrades everything else. A 2023 study measured this across models from 1 to 7 billion parameters during continual instruction tuning and found catastrophic forgetting throughout: domain knowledge, reasoning and reading comprehension all decline as new tasks come in. Within the range tested, severity increased with model size⁷. The practical implication is that your evaluation set has to cover what you do not want to lose, not just the task you are teaching.

Open weights are not an open foundation. You get the numbers and enough to run them. You do not get the corpus, the filtering recipe or the training logs, which is what would let you audit where a behaviour came from. The 2021 report already flagged that opacity as the central problem of the category¹.

The label became marketing. As of July 2026, practically every model a vendor ships is announced as a foundation model, including small models tuned for a single task. The phrase has stopped carrying information in commercial material. One test filters most of it: how many different things have been built on this model by people who do not work for the vendor?

The base moves under you. With a model served over an API, a generic alias can start pointing at a new version with no notice, and the behaviour of every prompt-level adaptation you built moves with it. Pinning the version is what separates an investigable regression from a lost week.

What it costs, in orders of magnitude

Public numbers are scarce and rarely comparable, but one of them is enough to give the shape of the bill. DeepSeek reported that the full training run for V3, a mixture-of-experts model with 671 billion total parameters and 37 billion activated per token, pretrained over 14.8 trillion tokens, consumed 2.788 million H800 GPU-hours⁸. At the rental prices of that period, the order of magnitude lands in the low millions of dollars. The report itself notes that the figure covers the official run and excludes prior research, discarded experiments and data.

On the other side of the asymmetry: a low-rank tune over a 7-billion-parameter model, with a few thousand examples, runs on one hourly-rented GPU and finishes in an afternoon. That bill lands in the tens of dollars. Prompting, the third route, costs the API call and nothing else.

That five-to-six order gap is the economic reason the category exists at all. If deriving cost the same as training, nobody would have needed a word for the base.

Where to start

1. Use the instruct model unless you have a written reason not to. The legitimate reasons are usually one of these: you are doing your own post-training, you need perplexity numbers, or the task is genuinely text completion.
2. Test the base’s tokeniser on your language before you choose. Count the tokens a real paragraph of yours produces. The gap between two families can exceed 30% on the same sentence, and you pay it on every call.
3. Try prompting before tuning. Most of the easy gain is there, and it is the only adaptation route you can revert in a deploy.
4. If you do tune, start low-rank. Full fine-tuning rarely wins enough to pay for its cost and its forgetting risk.
5. Build the evaluation set before training, covering what you do not want to lose. Without it you measure the new task and discover the degradation of everything else in production.
6. Pin the exact base version in production. A generic alias means behaviour changes without a deploy.

Step 5 is the one most teams skip, and it is the only one that catches the most common failure mode of deriving from a foundation model: the new task improves, everything else gets slightly worse, and nobody notices until a user complains about something that used to work.

Footnotes

1. Bommasani et al. (2021) coined the term for models trained on broad data at scale and adaptable to many downstream tasks, and argued that the resulting homogenisation makes the defects of the base inherited by every adapted model below it. ↩ ↩² ↩³

2. Brown et al. (2020) showed that the task can be specified by examples inside the input text itself, with no gradient update to the model. ↩

3. Hu et al. (2021) froze the pretrained weights and trained low-rank matrices injected into each layer: 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning of GPT-3 175B, with no added inference latency. ↩

4. Ouyang et al. (2022) tuned GPT-3 on human demonstrations and then with reinforcement learning over human rankings. Outputs from the 1.3B model treated this way were preferred to those of 175B GPT-3. ↩

5. Touvron et al. (2023) released Llama 2 as a collection of pretrained models from 7 to 70 billion parameters alongside the chat versions derived from them. ↩

6. Zhou et al. (2023) tuned a 65B model on only a thousand curated prompts and responses, with no reinforcement learning, and got answers rated equal to or better than GPT-4’s in 43% of the cases evaluated. Their conclusion is that almost all knowledge comes from pretraining. ↩

7. Luo et al. (2023) observed catastrophic forgetting during continual instruction tuning in models from 1 to 7 billion parameters, across domain knowledge, reasoning and reading comprehension, with severity increasing with size inside that range. ↩

8. The DeepSeek-V3 technical report (2024) records 2.788 million H800 GPU-hours for the full training run of a model with 671B total parameters and 37B activated per token, pretrained over 14.8 trillion tokens. ↩