LoRA Fine-Tuning Lab
Teaching a real model a new habit
Fine-tuning normally means changing every one of a model's weights. LoRA leaves all of them frozen and trains two thin matrices beside a few of them instead. Here that was done for real on Qwen2.5-0.5B-Instruct, the same model as the Model Internals lab: the first adapter taught it to answer every question like a pirate — by changing about one weight in a thousand. Slide the adapter in and out, replay the training run, and see why such a small change is enough.Adapter
What the model says
The model's own continuation, greedy (always the top token), with the adapter off and at the strength you chose.
—
—
First token of the answer
The top candidates for the first token of the answer at the chosen strength. Watch the probabilities shift as the adapter is added.
What to look for
At strength 0 this is exactly the original model: the adapter's second matrix started at zero, and the slider multiplies its contribution. Raise it and the answers first get shorter and plainer, then the voice arrives in the word choice — "yer", "be", "matey" — more than in the first word, which often stays " The" or " Read". None of the sample questions were in the training set, so what you see is a learned habit, not memorized answers.
Watch the facts, too. Mostly they survive, but not always: on "What is the capital of Spain?" the 8-bit model served here answers "Barcelona" at full strength, where the full-precision copy it was trained on still says "Madrid". The adapter has nudged a fact that was already close, and the small rounding differences of 8-bit weights tip it the other way — a real hazard of training on one copy of a model and serving another.
The training run, replayed
Loss is the model's average surprise at each next token of the pirate answers (cross-entropy, in nats). The dots are the checkpoints where the run was recorded; pick one, or press play.
What to look for
Step 0 is the untouched model. The loss falls fastest early on, but the first thing learned is the register of the answers, not the voice: by step 10 they have turned chatty and storybook ("of course", "to tell a tale"), sometimes stuck repeating a phrase. "Matey" arrives around step 20, and from there the voice settles while the curve flattens. Follow the Spain question through the run: in the middle it answers "Barcelona", and only by the end is it back to "Madrid" — fine-tuning disturbs nearby knowledge before it settles. The loss never reaches zero, and should not: a model that predicted these exact answers word for word would have memorized them rather than learned the voice. Each step is ordinary gradient descent (see the Gradient Descent lab), using the Adam optimizer on the adapter's numbers only.
Everything the adapter was taught
Every question-and-answer pair the adapter trained on, exactly as written. The model saw each question in the training format and was scored only on how well it predicted the answer, word by word.
Held out: the lab's sample questions
Used on the Adapter and Training run tabs, and never trained on.
What to look for
How little there is. The whole lesson is a single screen of short exchanges — a few thousand tokens, against the trillions of tokens the model saw in pretraining — yet it reliably changes how the model talks about anything. That is possible only because the adapter is not teaching facts, just a way of speaking that the model can already produce.
The data also explains the results. Filter for "matey" and "hearty": nearly every answer ends with one of them, which is why the address at the end of a sentence is the adapter's strongest habit. Filter for "Arr": only a handful of answers open with it, which is why the first token of an answer usually stays a plain word. And filter for "capital": four capital-city questions, all answered correctly, and nothing about Spain — so the slip to "Barcelona" was not taught; it is the adapter's general push leaking into a nearby fact.
How the set was written
The questions cover deliberately varied ground — science, cooking, geography, advice, small talk — so the only thing they share is the voice of the answers; anything else common to them all would be learned too. The answers are short, factually right and consistent in style, and each is followed by an end-of-text marker, so the model also learns to finish and stop. None of the sample questions on the other tabs appear here, which is what lets those tabs test a learned habit rather than recall.
Singular values: the frozen weight against the update
Every matrix is a sum of rank-1 pieces, one per singular value. The frozen matrix needs hundreds of them; the update the adapter learned has at most 8 by construction.
How much each layer changed
The size of each layer's update relative to its frozen weight, ‖ΔW‖ / ‖W‖, for every projection the adapter modifies.
What to look for
The frozen matrix's singular values decline slowly: no small set of directions holds most of it, which is why the full matrix needs all its numbers. The learned update is the opposite. Even though it was allowed 8 directions, most of its size sits in the first one or two — the finding at the heart of the LoRA paper: the change fine-tuning needs has a very low intrinsic rank. The per-layer chart shows that the updates are small next to the weights they modify — a few percent, about a tenth at most — and spread across every layer rather than concentrated in one. The merge check confirms the bookkeeping: adding B·A into W gives the same outputs as running it beside W, to rounding error.
A pretrained language model is a general-purpose engine; most real uses want it bent toward something specific — a house style, a reply format, a specialist vocabulary, a narrow task. Doing that by retraining the whole model is slow, memory-hungry and leaves a full-size copy for every variant. LoRA (Low-Rank Adaptation, Hu et al., 2021) is the technique that made customization cheap, and it is now the default way open models are fine-tuned. Its core idea fits in one line of linear algebra, covered in the Matrices lab.
1. What fine-tuning actually changes
Nearly all of a transformer's knowledge and skill lives in its weight matrices: grids of numbers that each turn one vector into another. In every layer, one matrix turns a token's state into a query (what it is looking for), two more into keys and values (what it offers), a fourth folds the result of attention back into the token's state, and three more make up the feed-forward block that does most of the per-token processing — seven matrices per layer. This model has 24 layers of these, about 494 million numbers in all. Ordinary fine-tuning nudges every one of them. The real cost is memory, not arithmetic: an optimizer such as Adam keeps a gradient and two running averages beside each trained number, so full fine-tuning needs several times the memory of the model itself — and the result is one more full-size copy to store and serve.
2. The update, factored
Training a weight matrix $W$ ($d_{out} \times d_{in}$) means learning a change $\Delta W$ of the same size. LoRA freezes $W$ and writes the change as the product of two thin matrices:
$$W' = W + \frac{\alpha}{r}\,B A, \qquad A \in \mathbb{R}^{r \times d_{in}},\; B \in \mathbb{R}^{d_{out} \times r},\; r \ll d$$
Read it as two steps. $A$ squeezes the input down to just $r$ numbers — it decides which few directions of the input matter for the new behavior — and $B$ expands those $r$ numbers back into a full-size correction. The rank $r$ is the number of independent directions the change can use: a rank-8 update can push the layer's output along at most 8 directions, however large the matrix. That restriction is what makes it small. For one 896 × 896 query matrix, $8 \times (896 + 896) = 14{,}336$ trainable numbers stand in for 802,816. Across the 48 matrices adapted here (query and value in all 24 layers), that is 540,672 numbers: about one in a thousand of the model's weights.
3. Starting from the original model
$A$ starts as small random numbers and $B$ starts at exactly zero, so $BA = 0$: at step 0 the adapted model is the original, and training can only move away from it as far as the data justifies. Each training step is ordinary gradient descent, except that only $A$ and $B$ are updated. The error signal still travels backward through every frozen layer — that is how the adapter learns what the rest of the network needs from it — but no frozen weight is ever changed, and nothing about them has to be stored for the optimizer. The factor $\alpha / r$ (here 16 / 8 = 2) is a fixed scale that keeps the size of the update comparable when the rank is changed, so the learning rate need not be re-tuned. The strength slider on the Adapter tab multiplies this same term, from 0 (the original) to 1 (as trained).
4. Why a low rank is enough
Pretraining has already built the model's knowledge and skills; fine-tuning mostly changes how it uses them — which patterns to favor, which register to answer in. That kind of change turns out to need very few independent directions. Aghajanyan et al. (2020) fine-tuned a language model through only a couple of hundred free parameters, projected into the full network, and recovered 90% of full fine-tuning's accuracy on a benchmark task. Hu et al. (2021) found that LoRA ranks as small as 1 or 2 matched much larger ones on several tasks. The Low rank tab shows the same thing on this adapter: it was allowed 8 directions per matrix, and in the matrix it changed most, a single direction carries most of the change.
5. The choices in practice
Three settings define an adapter. The rank sets its capacity: 4 to 16 is typical for a style or format, higher for harder tasks or larger data. The targets are the matrices that get an adapter: the original paper adapted only the attention query and value matrices, as this one does; current practice often adapts every matrix in every layer, buying more capacity for a still small cost. And the data matters most of all: a few hundred well-chosen examples of the desired behavior often suffice, because the adapter is learning a habit, not the knowledge underneath it. The server behind this page can load several adapters at once, each modifying any of the seven matrices in a layer; the selector above the tabs lists the ones available.
6. How this adapter was made
The served model stores its weights in 8-bit form, which can be run but not trained through. So the adapter was trained on the same model's full-precision release, in Rust with the candle library, on an ordinary 6-core CPU in under half an hour: 112 short question-and-answer pairs written in a pirate voice, 120 steps of 4 examples each, scored only on the answers. The server then runs the 8-bit model with the 32-bit adapter beside it. That is close to the arrangement QLoRA (Dettmers et al., 2023) made standard — a compressed frozen base with a small precise adapter, which brought fine-tuning of a 65-billion-weight model within reach of a single GPU — except that QLoRA also trains against the compressed base, which avoids the precision mismatch noted on the Adapter tab. The adapter file is about 2 MB; the model it modifies is about 675 MB.
7. Merge it, or keep it separate
Because $W + \tfrac{\alpha}{r}BA$ is just another matrix of the same shape as $W$, an adapter can be folded into the weights once training ends; the model then runs at exactly its original size and speed. Keeping it separate, as this server does, costs two small extra multiplications for each adapted matrix but has real advantages: one copy of the base model can serve many adapters — a different one per customer, per task or per request — each a few megabytes, switched in instantly. And the adapter can be applied at any strength, which is what the slider on the Adapter tab does.
8. What LoRA is good at, and what it is not
LoRA excels at behavior: a voice, a reply structure, a domain's vocabulary and conventions, a fixed task such as classifying or extracting. It is a poor way to add facts. Knowledge from pretraining is spread across the whole network, and a small adapter trained on a handful of statements tends to memorize those sentences rather than make the fact usable in new contexts — while, as the Spain question in this lab shows, it can disturb nearby facts the model already had. For knowledge, retrieval (fetching the relevant documents and placing them in the prompt) is usually the better tool; LoRA then shapes how the model uses what it is given.
Answer every question before checking.