Is fine-tuning worth it in your case?

Fine-tuning is worth it when three things hold at the same time: a well-written prompt has already failed, the failure is about behaviour rather than knowledge, and you can show the right behaviour in a few hundred reviewed examples. Miss one of the three and training solves the wrong problem while charging you for the right one.

The request nearly always arrives in the same shape: “we want a model trained on our data”. Translated, it usually means “we want the model to know what’s in our documents” — and that is knowledge, which is retrieval territory. What fine-tuning changes is the weights, and what weights hold is behaviour.

The second most common translation is “the model won’t follow our format”. That one can be training, but there is a cheaper rung first: putting examples inside the prompt. You test that in an afternoon. The other takes three weeks and ties you to one base model.

The six questions

Has a well-written prompt already failed? Not the prompt somebody sketched on Monday. The prompt with the role stated, the output shape described, the decision criterion spelled out and two or three worked examples inside it. This check looks like paperwork and it is the one that eliminates the most candidates. Skip straight to training and you will never learn which of the two would have solved it.

Is the failure about behaviour? Behaviour is format, tone, section order, what to do when the input is ambiguous. Knowledge is the number in your pricing table. The question separates the two before you spend a GPU hour, and the answer changes the remedy completely.

Can you show what right looks like in a few hundred examples? If you cannot write fifty examples of the ideal output, the problem is not the model: the specification does not exist yet. Writing the examples usually resolves the ambiguity, and sometimes resolves the whole case, because the same examples fit in a prompt.

Is there an eval set written before training? Without one you have no way of knowing whether the run helped. Looking at ten answers and feeling that it got better is noise wearing the clothes of a result, and it is how most projects declare victory.

Does the volume pay for the run? The saving from fine-tuning comes from shortening the prompt on every call. It scales with call count, and the call count of most internal applications is small. Do the arithmetic with your own numbers before going further: tokens cut per call, times calls per month, times price per token. An internal tool with 3,000 monthly calls that cuts 1,500 tokens of prompt saves 4.5 million tokens a month, something around ten dollars. The example set that makes that cut possible costs weeks of work.

Will someone own this in six months? A tuned model is infrastructure. It needs versioning, somewhere to be served from, and somebody who knows how to redo the run when the original author changes teams.

A “no” at any point ends it. That is not rigidity: each question describes a condition without which training cannot work, and the sequence runs from the cheapest thing to check to the most expensive.

What goes in through the weights and what doesn’t

The second question kills the most projects, and there is evidence behind it.

A controlled 2024 experiment varied the share of training examples carrying facts unknown to the model, in closed-book question answering. Examples with new facts were learned considerably more slowly than examples compatible with what the model already knew. And as they finally were learned, the model’s tendency to hallucinate grew linearly¹.

Read the shape of that failure again, because it is treacherous. You train, the fact resists. You train more, it goes in. And the model starts inventing with more confidence on questions that have nothing to do with your domain. The gain on your test set hides a regression on everything else, and you only find out if you measured outside the domain — which almost nobody does.

The head-to-head with retrieval had already been run the year before. A 2023 study measured unsupervised fine-tuning against RAG on knowledge-intensive tasks and found a consistent advantage for retrieval, both for knowledge the model had seen in pre-training and for entirely new knowledge². Read the scope: the tuning tested there was continued training over raw documents, which is exactly what most teams try first.

There is a borderline case that muddies things: domain vocabulary. Product names, internal acronyms, industry jargon. It sits between the two columns. The model learns to use the term, which is behaviour, but it does not learn what the term means today, which is knowledge. In practice, the vocabulary usually fits in the system prompt in under two hundred tokens.

How many examples, really

The answer depends on what you are teaching, and both ends of the range have been measured.

For behaviour, the floor is surprisingly low. LIMA tuned a 65B model on a thousand hand-picked prompts and responses, with no reinforcement learning, and the result followed format and task well enough to compete with commercial assistants in the authors’ human evaluation³. A thousand examples teach nobody medicine. They teach a model to answer the way the author of those thousand examples would answer.

For hard capability, the floor is much higher. A 2024 study compared LoRA against full fine-tuning on programming and mathematics, with roughly 100,000 instruction pairs, and still saw LoRA fall substantially short of full fine-tuning at the usual low ranks⁴. When what you want is new capability rather than style, the size of the set changes order of magnitude.

A scaling study from the same year helps calibrate expectations before you build the set. Testing models from 1B to 16B on translation and summarisation, the authors found that fine-tuning gains scale with base model size more than with pre-training data size, that adding parameters to the efficient method generally does not help, and that the optimal method depends heavily on the task and on how much data you have⁵. The practical implication is dull and useful: no recipe transfers. What worked for somebody else does not predict your case.

The rule that survives contact with production: five hundred examples reviewed one by one beat fifty thousand scraped. Growing the set later is easy. Throwing away a large bad set is expensive, and nobody throws it away.

The alternatives, in the order they solve things

A better prompt. Explicit instruction about format, role and decision criterion. It is the only rung you can fix in thirty seconds after discovering it was wrong.

Examples inside the prompt. A controlled 2023 comparison held fixed what these arguments usually leave uneven — same model, same number of examples, sizes from 125M to 30B — and measured how the two approaches generalised to challenge datasets. Both generalised similarly, with wide variation depending on model size and example count⁶. That result dismantles the strong version of both camps: few-shot fine-tuning is not inherently brittle, and in-context learning is not inherently robust.

Retrieval. Find the right document and put it in the context. It is the rung for anything that counts as knowledge, and the only one where updating means editing a file.

Switching base model. Frequently forgotten and sometimes the cheapest move. The model that failed your case in January may have a successor in July that gets it right with no examples at all, and prompts and indexes travel between models almost for free.

Where the decision goes wrong

Confusing “I didn’t like it” with “it can’t do it”. If the output is technically correct and you want a different style, that is a prompt. Fine-tuning enters when the style is hard to describe in words and easy to show in an example.

Training on another model’s output without review. The set inherits the errors and the run learns them as efficiently as it would learn the right pattern. What comes out is a model that is consistently wrong.

Measuring only inside the domain. The same 2024 LoRA study found the other side of the coin: the adapter learns less, but it also forgets less than full fine-tuning outside the target domain⁴. You only see that if your eval set includes tasks that are not yours. Without it, you cannot tell which of the two effects caught you.

Treating the decision as permanent. A run that pays for itself today can stop paying when token prices drop or when a better base model ships. Question six exists precisely because today’s right answer has an expiry date.

Letting the training set describe the past. A concrete and common case: a support team exports 20,000 resolved tickets and uses the agent’s reply as the ideal output. The set is large, cheap, and full of things nobody wants any more — the old policy, the tone of two years ago, the answer the customer complained about afterwards. The run learns exactly that, faithfully, and the symptom shows up as a model that answers well and answers wrong. A historical archive is raw material for review, not a training set.

What it really costs

Order-of-magnitude numbers, not a quote.

Data is where the money goes. A few thousand examples reviewed by somebody who knows the correct output consume weeks from a person who is usually expensive and does not have those weeks free. Budgeting the training without budgeting this is the most common estimation error in the whole subject.

Evaluation is written first and redone every run. Thirty to a hundred real cases with the output you would accept, plus a set outside the domain to catch regressions. It costs days and it is not optional.

Training is the smallest line. At the sizes most teams use, between 7B and 14B, an adapter run takes a few GPU hours. In July 2026, renting by the hour, that lands in the tens of dollars. Every failed attempt costs the same again, which still leaves the line small.

Maintenance is versioning, serving and re-evaluating. A prompt has none of that.

Migration is the line nobody puts in the spreadsheet. When a better base model ships, the tuning does not travel: you redo the run, and the evaluation with it. The per-line detail lives in what a fine-tuning run costs.

Always compare against what you would stop spending. If the tuning cuts 2,000 tokens from every prompt and you make a million calls a month, the arithmetic closes fast. If you make a thousand calls a month, it never closes — and in that case the right calculation is not “how much do I save”, it is “what does not doing this cost me”.

The two-week test

Before approving the project, run this. It costs two weeks and answers the question.

1. Write the eval set. Thirty to a hundred real cases with the output you would accept, decided before any training.
2. Run the baseline. The best prompt you can write, then the same prompt with three examples inside it.
3. Assemble the ideal context by hand for ten hard cases: paste in the right document, the right policy, the right example. If the model gets them right that way, the problem is retrieval and you are done.
4. Classify the failures that remain. Wrong content, wrong format, or wrong judgement. The three have different remedies.
5. Build 200 examples of the behaviour that failed and train an adapter. A cheap run answers whether the signal exists at all.
6. Measure twice, inside and outside the domain. If the gain inside came with a loss outside, you traded one problem for another.

Step 3 is what dismisses most projects, and it is the one people skip. It separates “the model can’t do this” from “the model wasn’t given what it needed”, and those two sentences lead to budgets an order of magnitude apart.

Footnotes

1. Gekhman et al. (2024) varied the fraction of examples carrying unknown facts in a controlled closed-book QA experiment, and measured a linear increase in hallucination as those examples were learned. ↩

2. Ovadia et al. (2023) compared unsupervised fine-tuning against RAG on knowledge-intensive tasks, with a consistent advantage for retrieval on both seen and entirely new knowledge. ↩

3. Zhou et al. (2023) tuned a 65B model on a thousand curated examples with no reinforcement learning, and use the result to argue that almost all knowledge comes from pre-training. ↩

4. Biderman et al. (2024) compared LoRA and full fine-tuning on programming and mathematics: the adapter trails inside the target domain at usual ranks, and preserves performance outside it better. ↩ ↩²

5. Zhang et al. (2024) tested scaling factors on models from 1B to 16B and found gains driven more by base model size than by pre-training data, with the optimal method varying by task and data volume. ↩

6. Mosbach et al. (2023) compared few-shot fine-tuning and in-context learning while controlling for model, example count and scale from 125M to 30B, and found both generalising similarly with large variation. ↩