Can you fine-tune on a consumer GPU?

Yes, with the base model quantised. QLoRA loads the weights at 4 bits and trains only a low-rank adapter at higher precision, which drops memory far enough for the tuning run to fit on a consumer card. The authors trained a 65B model on a single 48GB GPU while preserving 16-bit fine-tuning performance¹.

The method is LoRA with one change underneath: instead of freezing the base weights at their original precision, it freezes them at 4 bits. Since they were frozen anyway, storing them with less precision does not interfere with what training moves.

What changes is the boundary of what you can train without renting an eight-card node. Before getting here, check whether another method in the family solves your case with fewer moving parts, because quantisation adds an error surface that was not there before.

Where the memory goes

Training memory comes in three parts, and they disappear in different orders.

The first is optimiser state. In mixed-precision training with Adam, every trainable parameter carries a gradient and two moments, roughly 16 bytes per parameter. For 7 billion of them, that clears 100 GB. This is the part a low-rank adapter removes almost entirely, because only a few million parameters receive gradient.

The second is the model itself, which LoRA leaves alone. A 7B model at bf16 occupies 14 GB before any training. Once the optimiser leaves the arithmetic, this becomes the dominant part, and it is exactly where quantisation lands.

The third is batch activations, which grow with batch size and sequence length. Neither method attacks that part, and it is the one that most surprises people who costed the first two and still blew past the card.

Where quantisation enters

The point that confuses most: the 4 bits are a storage format, not a compute format. Weights sit in NF4 in card memory, but each block returns to bf16 the moment it joins a multiplication, and the arithmetic happens at full precision.

Gradients cross those dequantised weights to reach the adapter, without updating any of them. Only A and B move, and neither was ever quantised.

The paper introduces three pieces¹:

• NF4, a 4-bit data type designed for weights with a near-normal distribution. The authors describe it as information-theoretically optimal for that distribution, which differs from spacing levels uniformly the way a 4-bit integer would.
• Double quantisation, which quantises the quantisation constants themselves. Each block of weights stores a scale constant, and at small block sizes those constants become a non-trivial share of the total.
• Paged optimisers, which use unified memory to absorb the memory spikes that would otherwise kill a run on long sequences.

Choosing 4 bits was not arbitrary. Earlier work by the same authors studied the trade between bit count and model size at inference and argued 4 bits sits at the right point of that curve for most cases².

How much VRAM each size asks for

The table below is arithmetic rather than measurement: parameters times bytes, plus headroom for the adapter and its optimiser. Activations stay out because they depend on your batch and sequence length, and they routinely add several gigabytes.

model	weights at bf16	weights at NF4	practical range with QLoRA
7B	~14 GB	~4 GB	fits in 12 to 16 GB
13B	~26 GB	~7 GB	fits in 16 to 24 GB
34B	~68 GB	~17 GB	wants 24 to 40 GB
70B	~140 GB	~35 GB	wants 48 GB or more

The published reference point is 65B on a 48GB card¹. The 70B row above is consistent with it and is not the same measurement, so treat it as an estimate.

Compare against full fine-tuning to see the distance: that same 7B model would ask for the 14 GB of weights plus something over 100 GB of gradient and optimiser state, which fits on no consumer card, quantised or not.

QLoRA or LoRA

The choice is about hardware constraints rather than quality, and the criterion is direct: use LoRA if the base model fits on your card at bf16, and QLoRA when it does not.

Quantising costs something. Training slows down because every block gets dequantised at each step, and you add an approximation between the model you evaluated and the model you trained. Neither is worth paying with memory to spare.

The “does it fit” arithmetic is easy to get wrong, because almost everyone forgets activations. A 7B at bf16 is 14 GB of weights on a 24 GB card, which looks roomy until batch and sequence add their own gigabytes. If the margin fell below roughly 30%, quantising costs less than discovering the overflow midway through the third epoch.

There is a middle case that often gets skipped: loading the base at 8 bits. It loses less precision than 4 bits, halves the weight memory, and settles plenty of cases where 4 bits would be overkill³.

Running it the first time

1. Load the model at 4 bits with NF4 and bf16 compute. In current libraries that is loading configuration, not a change to the training loop.
2. Turn on double quantisation. The gain is small in proportion and comes free; there is no reason to leave it off.
3. Start at rank 16 with scaling 32, applying the adapter to the attention projections. Same starting point as a plain LoRA run.
4. Measure a baseline before training, using the quantised base model and the best prompt you have. Without it you cannot separate the effect of training from the effect of quantisation.
5. Cut sequence length before cutting batch size when memory runs out. Activations grow faster with sequence than with batch in most configurations.
6. Fold the adapter into the full-precision weights, not the quantised ones, when you go to serve. Adding onto 4-bit weights introduces error that was not present during training.

Step 6 produces the most silent bugs. The model keeps answering, only worse than it did in evaluation, and the cause does not look like a code error.

What to do once training ends

Training at 4 bits finishes and leaves a decision almost nobody plans for: what precision the model will serve at.

There are three paths, and they are not equivalent. The first is folding the adapter into full-precision weights and serving the result at bf16. You recover maximum quality and go back to needing 14 GB, which makes sense when training and inference happen on different machines.

The second is serving the quantised base with the adapter applied on top, without merging. It is closest to the training condition, and it is the only path that lets you swap adapters per request. It costs one extra operation per layer.

The third is quantising again after merging, producing a 4-bit model with the behaviour already baked in. It is the cheapest to serve and the riskiest, because the second quantisation lands on weights training never saw in that form. Evaluate before assuming it came out the same.

The error that shows up most is merging the adapter straight into the 4-bit weights. The operation does not fail and the model keeps answering, so nothing flags the problem until somebody compares against the evaluation and notices the gap.

When memory runs out

Some symptoms have predictable causes, and recognising them saves hours.

It blows up on the first step. The problem is the model, not the training loop. Check that quantisation actually applied — loading at 4 bits without the right configuration silently loads at 16 bits in some library versions.

It blows up after a few steps. Those are activations, and the variable is the longest sequence in the batch. Sorting examples by length or truncating the top of the distribution fixes it faster than shrinking the batch.

It blows up only during evaluation. Evaluation usually runs without gradients but at a larger batch, and generation accumulates a key-value cache. Lower the evaluation batch separately, leaving the training batch alone.

It blows up intermittently. That is the case paged optimisers exist for, and it is worth confirming they are enabled before touching anything else.

Where it fails

The paper’s evaluation is weak at its most quoted point. The number that circulates is Guanaco reaching 99.3% of ChatGPT’s performance on the Vicuna benchmark¹. That benchmark uses a model as judge and carries few prompts, which makes it poor ground for a parity claim. The authors discuss evaluation limitations in the text themselves.

Quantisation and teaching new capability compound badly. The adapter already learns less than full fine-tuning when the task demands a large move⁴, and quantising the base stacks another approximation on top. For style and format the effect is small; for teaching a new domain, measure before assuming.

Training gets slower. Dequantising block by block at every step has a cost, and it shows up in epoch time. You traded time for memory, which only pays when the alternative is not training.

Not every 4-bit load is QLoRA. Loading at 4 bits with a generic scheme is a different thing, and the result can be considerably worse. The format was chosen for the distribution of the weights, and that choice is what carries the claim of preserved quality.

The card needs decent bf16 support. On older hardware, dequantising to bf16 can be slow enough that the memory saving stops being worth it.

What it costs

GPU cost stops being the dominant line item, and that changes the arithmetic of the whole project. A 24GB card rented by the hour handles a 7B or 13B tuning run, and the paper reports their best family coming out of 24 hours on a single card¹.

What does not change is everything else. Curating the dataset, defining the evaluation and maintaining the model over time still cost the same, and still make up most of the work. Cheaper training raises the share of effort going into data rather than lowering total effort.

Footnotes

1. Dettmers et al. (2023) tuned a 65B model on a 48GB GPU with NF4, double quantisation and paged optimisers, reporting preservation of 16-bit fine-tuning performance. ↩ ↩² ↩³ ↩⁴ ↩⁵

2. Dettmers and Zettlemoyer (2022) studied the trade between bit count and model size and argued for 4 bits in most cases. ↩

3. Dettmers et al. (2022) showed 8-bit inference without degradation on large transformers, handling outlier activation dimensions separately. ↩

4. Biderman et al. (2024) found the adapter behind full fine-tuning on programming and mathematics at usual ranks. ↩