What is parameter-efficient fine-tuning?

PEFT, short for parameter-efficient fine-tuning, is the umbrella for methods that freeze most of a pretrained model and train only a handful of parameters. You pay a fraction of the training memory and keep megabytes per task instead of gigabytes, at quality close to full fine-tuning across a good share of cases.

The name describes a constraint rather than a technique. Someone saying they did PEFT has not yet said what they did: four families fit under it, with different mechanics and, more importantly in practice, different inference arithmetic.

The family everybody uses today is LoRA, to the point where the two terms get conflated. Keeping them apart matters, because the choice between families changes system design, and because the decision that comes before it — whether tuning solves your problem at all — still saves the most work.

The question that comes first

One paragraph on the step before this is worth spending, because most projects that reach PEFT should have stopped earlier.

There is a ladder of intervention, running cheapest to most expensive: a better prompt, examples in the prompt, retrieval, efficient tuning, full fine-tuning, continued pretraining. Each rung costs more and takes longer to undo. The common error is jumping from the first to the fourth because the first got twenty minutes of effort.

What separates the rungs is the kind of problem. If the model does not know a fact, training will not fix that reliably and retrieval will. If the model knows but answers in the wrong format, the wrong tone or the wrong convention, then training is the right tool, and that is where PEFT shines. Telling knowledge apart from behaviour saves the most weeks.

The second criterion is volume. Efficient tuning pays off when you hold enough examples of the behaviour you want, reviewed by somebody who knows what correct looks like. A few hundred usually suffice for style and format. Without that set, and without a plan to build it, no method in the family helps.

What they all share

Full fine-tuning a 7B model is not expensive because of the weights. It is expensive because of what rides along with each trainable weight during training. In mixed-precision training with Adam, every parameter drags a gradient and two optimiser moments, roughly 16 bytes per parameter on the usual accounting. For 7 billion of them that clears 100 GB, on top of the model’s 14 GB in bf16.

Every PEFT method attacks that number through one door: cutting how many parameters receive gradient. What differs between them is where the trainable parameters sit.

Where each family puts parameters

Adapters came first, in 2019. They are small modules inserted between transformer layers, with a projection that squeezes the dimension, a non-linearity, and another that expands back. On GLUE, the authors landed within 0.4% of full fine-tuning while training 3.6% of parameters per task¹.

LoRA adds two thin matrices to each chosen weight matrix instead of inserting a module into the path. The product carries the original matrix’s shape, so it can be folded into the weights before serving. Against 175B GPT-3 tuned with Adam, the authors reported ten thousand times fewer trainable parameters and three times less GPU memory².

Prefix tuning leaves the weights untouched and optimises continuous vectors that enter as keys and values in every attention layer. Training 0.1% of the parameters, the original work reached comparable performance with abundant data and better performance in low-data settings³.

Prompt tuning is the simpler version of the same idea: the trained vectors enter only at the input sequence, not at every layer. The result that gives the paper its name concerns scale — the gap to full fine-tuning shrinks as the model grows, and closes in the tens-of-billions range⁴.

Selective methods add nothing at all. BitFit trains only the bias terms already present in the model, and reported performance competitive with full fine-tuning on small and medium datasets⁵.

Multiplicative methods are the newer branch. (IA)³ learns vectors that scale activations, and the T-Few recipe built on it beat the state of the art on RAFT by 6 absolute points, at compute well below what solving the same task through prompt examples would cost⁶.

The axis that decides in production

PEFT comparisons tend to rank by trainable parameters, which is the least useful of the three metrics. It describes the cost of training, which happens once. The cost that repeats on every request, forever, is a different one.

family	what it trains	cost after training
adapters	new modules between layers	extra operations per layer
LoRA	two matrices per chosen weight	zero, once folded in
prefix tuning	vectors on keys and values	part of the window at each layer
prompt tuning	vectors at the input	part of the context window
selective	weights that already exist	zero
multiplicative	vectors scaling activations	one multiply per activation

The LoRA row explains its dominance. Because B·A carries the frozen matrix’s shape, you add once and serve a model with the original architecture. That was a stated goal of the paper, written as a direct contrast with the 2019 adapters².

The prefix and prompt tuning rows hide a cost that never shows up in a benchmark. The vectors occupy positions in the sequence, and those positions come out of the budget available for user content. In a system with a tight context, that is expensive in a way no parameter table reveals.

What changes when the tasks are many

The storage saving reads like bookkeeping until you have more than one task. Then it becomes an architecture decision.

Under full fine-tuning, each trained behaviour is a whole model. Five behaviours on a 7B model are five 14 GB files, five loaded processes and five times the GPU memory, because nothing shares weights. In practice that means picking few behaviours and justifying each one.

With a method that produces a separable artefact, the base model loads once and the artefact goes on top. Five behaviours become five files of tens of megabytes served from the same card. The arithmetic shifts from “how many models fit the budget” to “how many files fit on disk”, and the answer stops constraining the design.

Not every family lends itself to this equally. LoRA and adapters produce an artefact that applies over the base without touching the original weights. Selective methods, which change weights that already existed, produce a modified model, which makes keeping several variants from one copy harder.

That is the practical reason LoRA won the argument, more than any benchmark quality difference. It is the only one delivering a small artefact, zero inference cost once merged, and the option of not merging when you need to swap behaviour per request.

Where it fails

Parity belongs to the task, not to the method. The numbers from 2019 to 2021 come from GLUE classification and short generation. A 2024 study compared LoRA against full fine-tuning on programming and mathematics and found the adapter substantially behind at usual ranks, though it preserved out-of-domain performance better⁷. Extrapolating parity from one regime to the other is the most common error.

PEFT does not teach new knowledge better than full fine-tuning. Training fewer parameters restricts how much the model can move, and that restriction is the mechanism rather than a side effect. When the task demands a large move, the restriction becomes a ceiling.

Cross-method comparison is fragile. A 2023 survey notes that work in the area rarely shares a base model, dataset and evaluation protocol, which makes cross-paper comparison tables unreliable⁸. Treat any general ranking with suspicion and measure on your own case.

Less training memory is not little memory. The base model stays fully loaded. A 7B in bf16 occupies 14 GB before any gradient, and no PEFT method changes that. Cutting that part needs quantising the base.

How to choose

1. Start with LoRA. It is the sensible default: known quality, zero inference cost once folded in, and the largest base of tooling and documentation.
2. If the model is very large and the tasks are many, consider prompt tuning. The scale result favours exactly that scenario, and the per-task artefact gets smaller still.
3. If you cannot add anything to the architecture, look at selective methods. Training biases leaves the model graph untouched, which simplifies deployment into rigid stacks.
4. Always measure outside the target domain. Every PEFT method preserves the original model better than full fine-tuning, and none preserves it perfectly.
5. Only move up to full fine-tuning after exhausting rank and matrix coverage. That order costs an afternoon per attempt; the reverse costs weeks.

In practice, Hugging Face’s peft implements most of these families behind one interface, wrapping a model you already loaded. Switching methods means swapping a configuration object, which makes step 4 cheap enough to be routine rather than a project.

What it costs

The arithmetic that matters is training memory, and it changes by orders of magnitude. For a 7B model in bf16, full fine-tuning asks for the 14 GB of weights plus something above 100 GB of gradient and optimiser state. A rank-16 LoRA adapter on the attention projections carries around 8.4 million trainable parameters, which puts that second figure near 130 MB.

The model’s 14 GB stays put. That is why the practical line of “fits on a consumer GPU” is not decided by the PEFT method alone, but by combining it with the precision the base model is loaded in.

Footnotes

1. Houlsby et al. (2019) landed within 0.4% of full fine-tuning on GLUE while training 3.6% of parameters per task, using modules inserted between layers. ↩

2. Hu et al. (2021) reported 10,000 times fewer trainable parameters and 3 times less GPU memory on GPT-3, with no extra inference latency. ↩ ↩²

3. Li and Liang (2021) trained 0.1% of parameters and reached comparable performance with abundant data and better performance in low-data settings. ↩

4. Lester et al. (2021) showed the gap to full fine-tuning shrinking with model scale and closing in the tens-of-billions range. ↩

5. Ben Zaken et al. (2021) trained only bias terms and reported competitive performance on small and medium datasets. ↩

6. Liu et al. (2022) learned vectors scaling activations and beat the state of the art on RAFT by 6 absolute points with the T-Few recipe. ↩

7. Biderman et al. (2024) found LoRA behind full fine-tuning on programming and mathematics at usual ranks, with better out-of-domain preservation. ↩

8. Lialin et al. (2023) point to the lack of shared base models, datasets and evaluation protocols as an obstacle to direct comparison. ↩