Why does training so few parameters work?

Because the difference between a tuned model and the original one is low rank: it fits in a space with far fewer dimensions than the matrix it changes. If that holds, training the whole matrix wastes effort. Two thin matrices suffice, with their product carrying its shape.

The idea grew out of a deployment constraint rather than a theoretical question. Keeping an independent fine-tuned instance per task is prohibitively expensive on a model of 175 billion parameters¹.

What follows is the argument holding the method up, with the numbers that support it and the point where the evidence stops. Day-to-day configuration lives in the article on the method itself, and its place among the efficient tuning methods has its own piece.

Where the hypothesis came from

The notion that neural networks carry fewer degrees of freedom than parameters predates LoRA. In 2018 it was measured directly: training networks inside random subspaces of growing dimension, intrinsic dimension was named for the smallest subspace where the network still reaches acceptable performance².

By 2020 the same reasoning reached fine-tuning of language models, and the result is quantitative. Optimising only 200 trainable parameters, projected back into the full space, reached 90% of full fine-tuning performance on MRPC. Two observations from the same work close the argument: pretraining implicitly lowers intrinsic dimension, and larger models tend to have smaller intrinsic dimension³.

LoRA’s step was moving that hypothesis from the parameters to the change in them. If adaptation lives in a small subspace, then ΔW, the difference between the tuned matrix and the original, should be low rank.

It helps to name what that is: a bet with indirect evidence. The empirical check on rank deficiency came after the proposal, as verification rather than a premise proven in advance¹. Much of the later confusion about when the method works comes from treating the bet as a theorem.

Why not the methods that already existed

Efficient tuning was not empty ground in 2021. Adapters had been around since 2019 and prefix tuning had landed months earlier. Dismissing both took an argument, and that argument explains design decisions that look arbitrary today.

Against adapters, the problem is latency. The modules have to be processed sequentially, and that shows up even at small bottleneck dimensions. In online inference at batch size 1 on GPT-2 medium, latency ran 20.7% to 30.3% above LoRA’s¹. It worsens once the model is sharded across cards, because extra depth means more synchronous operations.

The condition matters and usually gets dropped. That cost appears in online, short-sequence, small-batch scenarios, where no hardware parallelism absorbs it. In large-batch training it vanishes. If your application is an overnight batch job, the criticism does not apply to you.

Against prefix tuning, the problem is optimisation. The method is difficult to optimise and its performance changes non-monotonically in trainable parameters: past 256 special tokens for prefix-embedding tuning, and past 32 for prefix-layer tuning, results degrade, most likely from a shift in the input distribution¹.

There is also a structural cost: reserving sequence positions for the prefix cuts what remains for the task. On the short-window models of 2021 that was expensive. The constraint stays true and weighs less on a long window.

The decomposition

The mechanical part fits on one line. Where there was W·x, there is now W·x + B·A·x, with W frozen, A shaped r × d and B shaped d × r. Only A and B receive gradient.

Two initialisation details let the method start without a jolt. A begins with random Gaussian values and B begins at zero, which makes the product exactly null on the first step. The model departs from the original only as far as the gradient asks.

The product gets multiplied by α/r, with α fixed at the first r tried and not retuned afterwards¹. The reason is practical: without that factor, doubling the rank would double the magnitude of the update, and every change of r would mean recalibrating the learning rate.

The numbers, at GPT-3 scale

The evidence covers RoBERTa in base and large, DeBERTa XXL, GPT-2 in medium and large, and 175B GPT-3. The tasks are the eight GLUE benchmarks for understanding, E2E NLG, WebNLG and DART for data-to-text generation, and WikiSQL and SAMSum on GPT-3¹. Keep that list in mind when reading any conclusion, because it bounds the reach of everything that follows.

The most quoted summary is the comparison against 175B GPT-3 tuned with Adam: ten thousand times fewer trainable parameters and three times less GPU memory, at equal or better quality on RoBERTa, DeBERTa, GPT-2 and GPT-3¹.

The number that changes a system design is a different one. At rank 4, GPT-3 ends up with 4.7 million trainable parameters, and what you store per task drops from 350 GB to 35 MB. A hundred adapted models come to 354 GB instead of 35 TB, because the base model counts once¹. In that configuration, the result was 73.4 on WikiSQL and 91.7 on MNLI-m.

The absence of extra latency is the other deliberate result, framed as a direct contrast with the 2019 adapters, which inserted new modules into each layer’s path⁴. Since B·A carries W’s shape, the addition can happen once before serving, and the resulting model has the original architecture.

What the rank measurement showed

This is the part that shows up least in summaries and helps most when configuring.

Sweeping ranks from 1 to 64 turned up something uncomfortable to accept: rank 1 already sufficed for adapting the query and value projections on those datasets¹. That is not a result about language models in general. It is a result about those tasks.

The explanation came from comparing the subspaces learned at different ranks using normalised Grassmann distance. Directions at the top of the spectrum overlap heavily between rank 8 and rank 64; the rest do not. The reading is that extra rank mostly captures noise, which explains why raising r bought so little in that regime.

The same analysis across two random seeds at rank 64 showed more shared directions in ΔWq than in ΔWv, suggesting higher intrinsic rank in the query projection.

At a fixed budget of 18 million parameters, splitting between query and value beat concentrating on a single matrix¹. That is the origin of the default every library still ships, and the justification is empirical rather than theoretical.

Where the evidence stops

It is from 2021. The experiments cover GLUE classification, short generation on E2E, and the three GPT-3 tasks. Nothing there resembles modern instruction tuning over hundreds of thousands of examples, and generalising from one regime to the other is exactly what went wrong later.

Quality did not hold across every domain. A 2024 study compared the method against full fine-tuning on programming and mathematics, in both instruction tuning and continued pretraining, and found the adapter substantially behind at usual ranks. The same work measured full fine-tuning learning perturbations of rank 10 to 100 times greater than typical configurations⁵. If the low-rank hypothesis held universally, that measurement would not exist.

Equivalence was never promised. The original claim is parity on the tasks tested. The common reading that LoRA equals full fine-tuning is an extrapolation by the community, and keeping that distinction straight saves the most frustration.

Rank 1 is not configuration advice. It was a measurement on a specific set. Starting at 1 today, on an instruction tune, produces a model that learned nothing and a wrong diagnosis about the dataset.

Folding the weights in and serving many tasks are goals in tension, and that limitation was stated from the start. Absorbing A and B into W to zero out latency makes batching inputs from different tasks in one forward pass non-trivial, because each would need a different W. The escape is not merging when latency is not critical¹. That tension opened the research line on serving many adapters at once, years later.

What survived

1. The decomposition. This is the contribution that became infrastructure, and nobody disputes it. Every efficient tuning library implements W + B·A.
2. Folding in before serving. It solved the latency problem that held back earlier adapters, and it remains why the method dominates in production.
3. The adapter as a separable file. It sat outside the main argument and became the most used consequence, because one base model can serve many behaviours.
4. The hypothesis, with a caveat. It holds as a useful approximation for style, format and tone. For teaching new capability, treat it as an open question and measure.

If you read the original, go for section 7, the rank investigation. You already know the proposal from any library; what documentation never carries is the evidence gathered to justify it and the care taken stating it.

Edward Hu, who led the work, recounts that all of it came out of a product question at Microsoft in 2021, with few-shot falling short of production and GPT-3’s checkpoint making task and customer switching impractical⁶. That origin explains the shape of the result: the optimisation target is deployment cost, which is why the absence of latency gets as much room as the quality numbers.

Footnotes

1. Hu et al. (2021) proposed the decomposition, reported a 10,000-fold cut in GPT-3’s trainable parameters and 3-fold in memory, measured subspace similarity across ranks, and stated the batching limitation. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹

2. Li et al. (2018) measured intrinsic dimension by training networks inside random subspaces of growing dimension. ↩

3. Aghajanyan et al. (2020) reached 90% of full fine-tuning performance on MRPC by optimising 200 parameters projected back into the full space, and observed that pretraining lowers intrinsic dimension. ↩

4. Houlsby et al. (2019) inserted new modules between layers, which adds operations to the inference path. ↩

5. Biderman et al. (2024) found the adapter behind full fine-tuning on programming and mathematics at usual ranks, and measured perturbations of rank 10 to 100 times greater under full fine-tuning. ↩

6. Hu traces the method to a product question about GPT-3, and frames LoRA as a generalisation of full fine-tuning. ↩