What is LoRA in fine-tuning?

LoRA, short for low-rank adaptation, freezes the model’s weights and injects into each layer two small matrices that get trained in their place. You tune roughly 0.1% of the parameters and get back a file of tens of megabytes rather than a whole model. It is the default fine-tuning method outside labs that own hundreds of GPUs.

The problem it solves is logistical before it is about quality. Tuning every weight of a 7B model means holding, alongside the weights, a gradient and an optimiser state for each of them. That arithmetic overruns the memory of any card that is not a data-centre part, and what you get for it is one whole model to version and serve per trained behaviour.

LoRA is the most used family inside PEFT, the umbrella of efficient methods, and it has variants that squeeze memory further, such as QLoRA. The original idea, published in 2021, fits in a paragraph — and the reason it works is more interesting than the mechanics.

What training moves

Every transformer layer carries large weight matrices. In an attention layer of a model with 4,096 dimensions, the query projection matrix is 4,096 by 4,096, or 16.8 million numbers.

Full fine-tuning changes all 16.8 million. LoRA touches none of them. It adds two matrices instead: one projecting the 4,096 dimensions down to a much smaller space of rank r, and another projecting back up. With r = 16 they add up to 2 × 16 × 4,096, roughly 131 thousand numbers — 0.8% of the original matrix.

Their product has exactly the shape of the frozen matrix, and is added to its output. As far as the rest of the network is concerned nothing changed shape: the layer takes the same input and returns something the same size, only displaced by the adapter’s contribution.

Two initialisation details matter more than they look. The first matrix starts with small random values and the second starts at zero, which makes the product exactly zero on the first step. The model begins identical to the original and the adapter only drifts from it as far as the gradient asks. And the result is multiplied by a scaling factor tied to the rank, so that changing r does not also change the magnitude of the adjustment.

At serving time the addition can be done once and written into the weights. The resulting model runs exactly like the original, with no extra layer in the path — which is the practical difference from earlier adapter methods, which added modules and, with them, latency¹.

Why training so little works at all

The obvious question is why 0.1% of the parameters would be enough. The answer started to appear before LoRA existed.

A 2020 paper investigated why a model with hundreds of millions of parameters can be tuned on a few thousand labelled examples without everything falling apart. The answer they propose is intrinsic dimension: there exists a low-dimensional reparameterisation as effective for fine-tuning as the full parameter space. The number they measured is startling — optimising only 200 trainable parameters, randomly projected back into the full space, reached 90% of full-parameter performance on MRPC².

Two further observations from the same work close the argument. Pre-training implicitly minimises intrinsic dimension, and larger models tend to have lower intrinsic dimension after a fixed number of updates. The bigger the model, the fewer degrees of freedom adaptation needs.

The LoRA authors took that hypothesis and applied it to the change in weights rather than to the weights themselves: if adaptation lives in a low-dimensional space, the difference between the tuned model and the original should be low-rank too. The empirical investigation of rank deficiency in the paper itself is what backs the choice¹.

The file size is the point

The original paper’s numbers are from another scale and help pin the order of magnitude. Compared with 175B GPT-3 fine-tuned with Adam, LoRA cut trainable parameters by ten thousand times and GPU memory by three times, at equal or better quality on RoBERTa, DeBERTa, GPT-2 and GPT-3¹.

Redo the arithmetic at a size you might actually use. A 7B model with 32 layers and 4,096 dimensions, with rank-16 adapters on the query and value projections, has around 8.4 million trainable parameters. That is 0.12% of the total.

The memory saving does not come from the weights, which stay loaded. It comes from what accompanies the trainable ones. In mixed-precision training with Adam, every trainable parameter drags along a gradient and two optimiser moments — on the usual accounting, something like 16 bytes per parameter. For 7 billion of them that is over 100 GB, on top of the 14 GB of the model in bf16. For 8.4 million, it is about 130 MB.

That difference is what turns a project from “we need a node with eight cards” into “it fits on the card we already have”. QLoRA pushed the boundary one rung further by quantising the base model to 4 bits: the authors tuned a 65B model on a single 48GB GPU, and the best family they released came out of 24 hours on one card³.

What rank to use

The short answer is: start at 8 or 16 and climb if your eval set asks. The long answer is more interesting, because the right rank depends on what you are teaching.

For style, format and vocabulary, low ranks usually suffice, and it was that regime that made the method popular. For teaching new capability, the recent evidence is less cheerful. A 2024 study compared LoRA and full fine-tuning on programming and mathematics, both in instruction tuning with roughly 100,000 pairs and in continued pre-training on 20 billion tokens, and found LoRA substantially behind at the usual ranks⁴.

The same work carries the measurement that helps most when picking r: full fine-tuning learns perturbations with a rank 10 to 100 times greater than typical LoRA configurations. If your task demands changing the model a lot, an r of 8 is too small by construction, and no number of epochs fixes that.

The good side of the same study: the adapter forgets less. Outside the target domain, LoRA preserved the base model’s performance better than full fine-tuning, and better than common regularisers like weight decay and dropout⁴. Less new capability, less damage to what already worked. For most production cases that is a good trade.

Where to put the adapter

The second configuration decision is which matrices to inject the two small ones into, and it interacts with rank.

The original paper applied them to the query and value projections in attention, and that is still the library default. The justification is economic before it is theoretical: those were the matrices where the same parameter budget bought the most in their experiments¹. Later work extended it to every linear projection, including the dense feed-forward layers, which hold most of a modern transformer’s weights.

The rule that holds in practice: applying to more matrices at a lower rank tends to beat applying to few at a high rank, for the same trainable parameter count. That fits what the 2024 study measured — if the change your task demands is high-rank, concentrating it in two matrices squeezes it further still.

The file stays small either way. The 8.4 million parameters from the earlier example, stored in bf16, come to about 17 MB. Extended to every linear projection, something around 80 MB. Against the base model’s 14 GB, the difference between the two configurations does not matter for storage. It matters for quality.

One base model, many behaviours

The architectural consequence changes system design the most, and shows up in introductory explanations the least.

Because the adapter is small and separable, you can keep one copy of the base model on the GPU and swap the adapter per request. Five different behaviours become five files of tens of megabytes, not five models of 14 GB.

That became a systems problem in its own right. A 2023 paper treated adapter memory as paging, keeping all of them in main memory and fetching to the GPU only those the running queries need, in a unified pool shared with the key-value cache. With that, they served thousands of adapters on one GPU, with up to 4 times the throughput of the libraries available then⁵.

What this buys in practice is granularity. One adapter per customer, one per language, one per document type — decisions that would be economically absurd if each demanded its own separately served model.

Where it fails

A rank too small for the task. It is the most common failure mode and the easiest to misdiagnose, because it looks like a shortage of data. If training loss stops falling early and adding examples does not help, try a larger r before touching the set.

Apparent equivalence with full fine-tuning. A 2024 study analysed the weight matrices produced by both methods through singular value decomposition and found different structures. Those trained with LoRA gain new, high-ranking singular directions the authors call intruder dimensions, which do not appear with full fine-tuning. Intervening on them causally, they showed those directions cause the forgetting — and that scaling them down recovers much of the pre-training distribution performance with a small drop on the task⁶.

Accumulation across sequential training. The same work shows those dimensions pile up when you tune repeatedly in sequence, and that LoRA models do worse in that setting⁶. If your plan is to stack an adapter on an adapter over the months, that is the problem you will meet.

Misconfigured scaling. The factor multiplying the adapter is set relative to the rank. Changing r without changing the factor, or copying another project’s configuration, produces an adjustment that is too weak or unstable, and the symptom does not look like a configuration error.

Merging onto a quantised model. If you trained with the base in 4 bits and plan to merge the adapter into the weights, merge onto the full-precision version. Merging onto the quantised one introduces error that was not there during training.

Configuring the first run

1. Start with r = 16 and scaling of 32, applying the adapter to the attention projections. It is the most common starting point and works for most behaviour work.
2. Measure the baseline first, with the base model and the best prompt. Without it there is no way to know whether the adapter helped.
3. Run one to three epochs over a few hundred reviewed examples. More epochs on little data produce memorisation, not generalisation.
4. If there is no signal at all, raise the rank before growing the set. It is the cheaper change and it fixes the case where the task needed more adjustment capacity than r allowed.
5. Extend the adapter to the dense layers if the higher rank still is not enough. It costs more and usually buys more than raising r indefinitely.
6. Evaluate outside the domain. The adapter forgets less than full fine-tuning, but it does not forget nothing, and only the outside set shows how much.

Step 4 saves the most time. When tuning does not take, the instinct is to double the training set, which costs weeks. Changing one number in a config costs an afternoon and answers the same question.

Footnotes

1. Hu et al. (2021) report, against 175B GPT-3 fine-tuned with Adam, a 10,000-fold reduction in trainable parameters and a 3-fold reduction in GPU memory, with no added inference latency. ↩ ↩² ↩³ ↩⁴

2. Aghajanyan et al. (2020) measured the intrinsic dimension of fine-tuning and reached 90% of full-parameter performance on MRPC by optimising just 200 parameters projected back into the full space. ↩

3. Dettmers et al. (2023) fine-tuned a 65B model on a 48GB GPU by quantising the base to 4 bits and training low-rank adapters over it. ↩

4. Biderman et al. (2024) compared LoRA and full fine-tuning on programming and mathematics: the adapter trails at usual ranks, forgets less outside the domain, and full fine-tuning learns perturbations of 10 to 100 times greater rank. ↩ ↩²

5. Sheng et al. (2023) kept adapters in main memory and paged them to the GPU on demand, serving thousands of them on one card with up to 4 times the throughput. ↩

6. Shuttleworth et al. (2024) found, in the singular value decomposition, new directions that appear only with LoRA, showed by causal intervention that they cause the forgetting, and that they accumulate across sequential runs. ↩ ↩²