Model Internals Lab
Inside a real language model
Every number on this page comes from a real model — Qwen2.5-0.5B-Instruct, 24 layers and about half a billion weights, stored in 8-bit form — run by our own Rust code. Pick a prompt and look at three things no chat window shows you: how sure the model is about the next word, how that guess forms layer by layer, and which earlier words each word is paying attention to.Next-token prediction
The model's top 10 candidates for the very next token, out of 151,936, with their probabilities.
What to look for
A language model never "knows" the next word — it produces a probability for every token in its vocabulary, and a chat app then picks one. Some prompts leave the model confident ("one, two, three, four," gives " five" 89%), others spread the bet ("…watched the" — many endings are fine). Entropy measures that spread: near 0 means certain, higher means more options are still alive. Notice too that tokens are not words: " Paris" carries its leading space, and digits are split one per token.
Logit lens
Each cell reads the model's state at one layer and token as if it were finished: its top guess for the next token, brighter = more confident. Read a column bottom to top to watch a prediction form. Click any cell.
What to look for
The bottom row ("embed") just echoes each token — this model reuses one table for reading and writing words, so before any layer has run, every token "predicts" itself. The middle layers show junk: punctuation, code fragments, other languages. The real answer typically surfaces only in the last handful of layers (" Paris" at layer 22 of 24). The chart shows the same story for the last token, plus the length of the hidden vector, which grows from under 1 to about 70 — the reason every block starts by normalizing it (see Sphere & Rotations).
Attention
Rows are the tokens doing the looking; columns are the earlier tokens they look at. Brighter = more attention. Each row adds up to 100%. Click a row to read it.
What to look for
Each of the 24 layers has 14 heads, and each head computes its own pattern. Load "Code: 74928. Repeat the code: 749" and press the induction-head button: layer 10, head 13 sends all of the last token's attention to the "2" that followed the earlier "9" — it has found the pattern and points at what comes next, and the model predicts "2" at 94%. Many heads instead pile their attention on the very first token whatever it is; this attention sink is a resting place, not a sign of meaning, so be careful reading single heads. The "avg" view blends all 14 heads of a layer.
Interpretability is the work of reading what a trained network actually computes. The three views here are among its basic tools, and each one is built from math covered in the Math Labs.
1. Where these numbers come from
The model is Qwen2.5-0.5B-Instruct (Apache-2.0 licence), stored in the GGUF format with 8-bit weights (Q8_0: blocks of 32 numbers sharing one scale). It runs on a CPU in Rust, using the candle library with our own copy of the model code, changed to record the hidden state after every layer and every attention pattern. The results were checked against llama.cpp on the same file: same tokens, same top prediction on every test prompt, probabilities within a few points. The saved examples on this page were recorded with that code; nothing is simulated.
2. The residual stream
Each token is carried through the network as one vector of 896 numbers, the residual stream. Every layer reads it, computes something (attention, then a feed-forward block) and adds its result back. Nothing is overwritten; information accumulates. That is why the stream's length grows layer after layer, and why each block normalizes its input first.
3. The logit lens
At the end, the model turns the last vector into word scores with one normalization and one matrix: $\text{logits} = W_{\text{out}}\,\text{norm}(h)$. The logit lens applies that same final step to the stream after every layer, as if the model stopped early. It is an approximation — middle layers were never trained to be read this way — but it shows clearly that answers usually form late, after the earlier layers have assembled context.
4. Attention heads
Each head scores every earlier token with a dot product of query and key vectors, turns the scores into weights with a softmax, and averages the earlier tokens' values with those weights. Some heads have clean, nameable jobs. An induction head finds an earlier copy of the current token and attends to whatever followed it — the core of in-context copying. Others track the previous token, or punctuation, or rest on the first token as an attention sink. Heads work in teams, so a single heatmap is a clue, not a full explanation.
5. What these views cannot tell you
Attention shows where information can flow, not what it means or whether the model relied on it. The logit lens is a lens, not a readout of intent. Real interpretability research goes further: switching parts off (ablation), patching activations between prompts, and finding directions in the residual stream that stand for concepts. Those need the live model, which is the next step for this lab.
Self check
Answer every question before checking.