What is supervised fine-tuning?

Supervised fine-tuning, or SFT, means continuing the training of a finished model on pairs of prompt and ideal response. The objective stays what it was during pre-training — predict the next token — but the error is measured only over the response tokens. It is the most common form of fine-tuning and the step that turns a base model into one that answers requests.

The “supervised” in the name comes from that: somebody wrote the correct answer. That is the difference from pre-training, which only sees running text and has no answer key. And it is the difference from alignment methods, which start from a comparison between outputs rather than from a single right answer.

Practically every model you reach through an API went through this step. The foundation model that comes out of pre-training does not know it is supposed to answer: it knows how to continue text. SFT is the first rung that teaches the difference, and it is also the most approachable rung when you fine-tune something yourself.

The pair, and where the loss lands

One row of the training set has two parts: what a user would write and what you want back. In practice that is usually one line of JSONL per example:

{"messages": [
  {"role": "system",    "content": "Classify the ticket. Reply with the JSON only."},
  {"role": "user",      "content": "I was charged twice on the March invoice."},
  {"role": "assistant", "content": "{\"type\": \"billing\", \"urgency\": \"high\"}"}
]}

The three parts are concatenated into a single sequence, with markers saying where one ends and the next begins — the same markers the API uses to separate roles in a conversation. What the trainer sees is not a structure with fields: it is text with delimiters, start to finish.

The model reads the whole sequence and, at each position in the response, tries to predict the next token from everything that came before. It misses by some margin, that margin becomes a gradient, and the weights move a little in the direction that would shrink the error. The mechanism is identical to pre-training. What changes is the cut.

That cut is the most consequential engineering decision in the step. Prompt tokens are normally masked out: they enter as a condition for the prediction but do not count as targets. The reason is blunt — you do not want the model to get better at generating user questions. You want it to get better at answering them.

Two more choices round out the recipe, and both are conservative for good reasons. The learning rate is small, typically an order of magnitude below what pre-training used, because the point is to adjust a good model rather than rewrite it. And the number of passes over the set usually sits between one and three: past that, the model starts memorising the examples instead of learning the pattern they share.

The sign that you went too far is recognisable. Training loss keeps falling while validation loss flattens or climbs, and the model starts reproducing verbatim chunks of the examples in situations that never called for them. If you did not hold out a validation slice before starting, you cannot watch that happen — and the first report arrives from a user puzzled by an answer that reads like a copy.

There is an implementation detail worth knowing because it changes results. Short examples get packed together until the training window is full, so compute is not wasted on padding. If the packing does not isolate examples in the attention mask, the model reads the end of one example as context for the next and learns transitions that do not exist. Good libraries handle this; it is worth checking that yours does.

From base model to instruction follower

The clearest demonstration of what SFT does is to hand the same input to both models.

Write “list three causes of churn” to a base model. A plausible response is another list of similar questions, because somewhere in the training corpus that sentence appeared inside a list of questions. The model is not failing. It is doing what it always did: continue the text in the most likely way.

The same request to the model after SFT returns three causes of churn. Nothing was added in capability. What changed is the model’s expectation about what follows a sentence of that shape, and that expectation came from a few thousand examples where what followed was the answer.

The formulation is worth keeping, because it heads off an expensive confusion: SFT does not make the model more capable, it makes the capability reachable through the channel you use. A base model can solve the task if you write the prompt in the right continuation format. Tuning removes the need for that trick.

What instruction tuning proved

The most studied special case of SFT is instruction tuning: instead of training for one task, you train across many different tasks described in natural language, hoping the model generalises to tasks that were not in the set.

The first big result is from 2021. A 137B model was tuned on more than 60 language-processing tasks, each expressed as an instruction, then evaluated on task types held out of training. It beat zero-shot GPT-3 at 175B on 20 of the 25 tasks evaluated¹. The smaller model won because it had been taught to understand the request.

The same year, a parallel line of work approached it from the other side. Converting a large collection of supervised datasets into prompted form, with varied wordings for each, the authors tuned an encoder-decoder model and measured performance on completely held-out tasks. It beat models up to 16 times its size on several standard datasets².

In 2022 came the version at scale. Tuning across 1,800 tasks, including chain-of-thought data, a 540B model gained 9.4 percentage points on average over its untuned counterpart, and reached 75.2% on five-shot MMLU³. The authors measured three axes separately: number of tasks, model size and the presence of reasoning data in the set. All three matter.

And there is the result pulling the other way, which is the most useful of all for anyone about to build a set. LIMA tuned a 65B model on a thousand hand-selected prompts and responses, with no reinforcement learning at all, and the result competed with commercial assistants in the authors’ human evaluation⁴. A thousand curated examples against hundreds of thousands scraped. The conclusion they draw is strong and deserves careful reading: almost all the knowledge comes from pre-training, and tuning mostly teaches which format to use.

To mask the prompt or not

This is the configuration line almost nobody revisits, and it now has recent evidence against the autopilot.

The default is to mask: loss falls on the response only. A 2024 study tested the opposite, applying loss over the instruction as well, across 21 benchmarks. The gain showed up in two specific settings — sets with long instructions and short outputs, and small sets, of the kind LIMA popularised. In the most favourable case they measured, performance on AlpacaEval 1.0 more than doubled⁵.

The explanation the authors offer is overfitting: with short responses and a small set, there is little signal in the response half and the model memorises the set. Adding the instruction to the calculation gives more tokens to pull gradient from.

Do not read that as “stop masking”. Read it as: the decision depends on the shape of your set, and the shape of your set is measurable in five minutes. Divide the average instruction length by the average response length and count the rows. If the ratio is high and the rows are few, both configurations are worth a run.

Where SFT fails

The set becomes the product, and it is usually bad. If the outputs in your set are inconsistently formatted, tuning learns the inconsistency as efficiently as it would learn the right pattern. One bad row in ten teaches the model to be bad one time in ten.

New facts go in badly. SFT is excellent for behaviour and poor for knowledge that changes. That is the same limitation fine-tuning has in general, and the remedy is the same: retrieve the document instead of training on it.

A single correct answer is a convenient fiction. For “classify this ticket” there is an answer key. For “write a reply to this customer” there are dozens of acceptable answers, and SFT forces you to pick one and call it the target. That limitation is exactly what motivates alignment methods: instead of one right answer, they start from a comparison between outputs. InstructGPT made this concrete by showing that a 1.3B model aligned that way had its outputs preferred over those of 175B GPT-3 in their human evaluation⁶.

Regression outside the domain. Training on a narrow distribution degrades performance outside it. The model gets excellent at classifying your tickets and worse at writing anything else, and that only surfaces if your eval set includes tasks that are not yours.

The wrong chat format. Every model family has its own role markers. Training with one format and serving with another produces a model that looks worse than it is, and the symptom does not resemble a configuration error.

What one run costs

Order of magnitude, to size the project, not a quote.

Compute is the smallest part. Five hundred examples of a thousand tokens make half a million tokens per epoch. At the usual sizes, between 7B and 14B, with adapters, that is minutes to a few hours of a rented GPU — tens of dollars in July 2026, failed attempts included.

The set is where the budget goes. Five hundred examples reviewed by somebody who knows the correct output take one to three weeks from a person who rarely has those weeks free. If the output needs judgement, like a customer reply, the number doubles, because every example needs a second pair of eyes.

Evaluation costs more than it looks because it is redone every run, and runs tend to number five or six before the configuration settles.

The practical arithmetic: the training fits on any team’s credit card, and the rest does not fit in most teams’ calendars. That is why SFT projects die from lack of people, not lack of GPU.

Building the first run

1. Write 30 evaluation cases before anything else, with the output you would accept. Without them the run cannot be judged.
2. Run the base model with the best prompt you can write and keep the result. Everything gets compared against that number.
3. Build 200 to 500 examples reviewed one by one. Whoever reviews needs to know the right answer. The set is the work, not the training.
4. Standardise the output before training. If half the examples answer in prose and half in a list, the model learns to flip a coin.
5. Use the base model’s chat format, the same one you will use in production.
6. Train with adapters first, one to three epochs. If an adapter does not move the needle, full fine-tuning rarely does either.
7. Measure inside and outside the domain. The second set is what reveals the hidden price of the first.

Step 4 separates a run that works from one that does not more than anything else, and it is the most tedious. Standardising five hundred outputs takes days and does not feel like machine-learning work. It is the machine-learning work.

Footnotes

1. Wei et al. (2021) tuned a 137B model on more than 60 tasks expressed as instructions and measured performance on held-out task types, beating zero-shot 175B GPT-3 on 20 of 25 tasks. ↩

2. Sanh et al. (2021) converted supervised datasets into prompts with varied wordings and showed zero-shot generalisation to completely held-out tasks, beating models up to 16 times their size. ↩

3. Chung et al. (2022) scaled instruction tuning to 1,800 tasks and measured three separate axes — task count, model size and chain-of-thought data — with a 9.4-point average gain at 540B. ↩

4. Zhou et al. (2023) tuned a 65B model on a thousand curated examples with no reinforcement learning, arguing that tuning teaches format more than knowledge. ↩

5. Shi et al. (2024) applied loss over the instruction as well across 21 benchmarks, with the gain concentrated in sets with long instructions and short responses, or few examples, attributing it to reduced overfitting. ↩

6. Ouyang et al. (2022) started from annotator-written demonstrations and then trained on rankings between outputs; the resulting 1.3B model had its outputs preferred over 175B GPT-3 in their human evaluation. ↩