Don Stephenson Don Stephenson

Math Labs — Introduction

The low-level machinery underneath a language model

A large language model looks like magic from the outside, but inside it is built from a surprisingly small set of mathematical moves, repeated billions of times: vectors and their angles, matrices that transform them, weighted sums, sines and cosines, and derivatives that say how to improve. Each lab in this section isolates one of those moves in two or three dimensions, where you can see it and drag it. Put together, they are the vocabulary you need to read how an LLM actually works.

Below: first, one pass through a language model, step by step, with the labs that show each step; then each lab on its own, with the one mechanism it demonstrates and where that mechanism lives inside an LLM.

1. One pass through a language model

When a model reads your prompt and predicts the next word, every token travels through the same short pipeline, repeated in dozens of stacked layers:

  1. 01

    Words become vectors

    Each token is looked up as a list of a few thousand numbers — a point in a high-dimensional space, written as coordinates in a learned basis.

    Basis FunctionsMatrices
  2. 02

    Position is added

    Word order is encoded with sines and cosines at many frequencies, or by rotating pairs of coordinates through angles that grow with position (RoPE).

    Fourier SeriesUnit Circle
  3. 03

    Vectors are normalized

    Before each block, every vector is rescaled to a standard length (then reweighted coordinate by coordinate), so its direction carries most of the meaning — close to a point on a high-dimensional sphere.

    Sphere & Rotations
  4. 04

    Attention compares tokens

    Matrices project each vector into a query, a key and a value. Dot products — cosines of angles — score how related two tokens are, and a softmax turns the scores into an average of the values.

    MatricesUnit CircleAffine Transforms
  5. 05

    A feed-forward layer reshapes

    Each vector goes through $Wx + b$, a nonlinearity such as ReLU or GELU, and another $Wx + b$ — affine maps building a function out of hinges.

    Affine TransformsBasis Functions
  6. 06

    Scores become a next word

    A final matrix turns the last vector into one score per word in the vocabulary, and a softmax turns those scores into probabilities.

    Matrices
  7. 07

    Training nudges every number

    Learning means taking the derivative of the error with respect to every weight and stepping downhill — a first-order Taylor approximation, applied billions of times.

    Taylor Series

The whole attention step from 04 fits in one line, and every symbol in it is covered by these labs: $$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d}}\right)V$$ $Q$, $K$ and $V$ are matrix products of the token vectors (Matrices); $QK^{\mathsf T}$ is a table of dot products, each one a scaled cosine of an angle (Unit Circle, Sphere & Rotations); and multiplying the softmax weights by $V$ is a weighted average whose weights sum to 1 (Affine Transforms).

2. What each lab contributes

In the order they appear in the menu:

Basis Functions Shows: any signal is a weighted sum of simple pieces, and the choice of pieces decides what is easy to describe. hidden states are coordinates in a learned basis, concepts behave like directions, and a ReLU network literally builds functions out of hinges. Unit Circle Shows: cosine and sine as the coordinates of a turning point, and rotation as a 2×2 matrix. a dot product is a cosine of an angle, the core of every attention score; rotary position encoding (RoPE) rotates pairs of coordinates exactly like this. Sphere & Rotations Shows: a direction versus a full orientation, direction cosines, and why rotation matrices preserve lengths and angles. normalization layers keep vectors on a sphere so direction carries meaning, cosine similarity compares them, and rotations (RoPE) move vectors without changing their length. Matrices Shows: a matrix as a transformation of space, and matrix–vector products as the basic operation. nearly all the computation is matrix multiplication — the query, key and value projections, the feed-forward layers and the final vocabulary scores. Affine Transforms Shows: linear maps plus a shift, homogeneous coordinates, and why batching everything into one matrix product is fast. every layer computes $Wx + b$; the bias lets boundaries sit anywhere; attention outputs are weighted averages; and batching tokens into matrix products is why GPUs can run models at all. Taylor Series Shows: approximating a function near a point by matching its value, slope and curvature. training follows the first-order (slope) approximation of the error downhill, and fast approximations of activations such as GELU come from the same idea. Fourier Series Shows: building any repeating shape from circles spinning at different frequencies. the original Transformer encodes each position as sines and cosines at geometrically spaced frequencies — a Fourier-style fingerprint of "where am I in the sentence". Convolution Shows: sliding a small set of weights along a signal, multiplying and summing. the same sliding sum powers image and audio encoders that feed multimodal models, and long-convolution and state-space layers (Hyena, Mamba) that compete with attention. DFT & Spectrum Shows: measuring the frequencies inside sampled data, windows and aliasing, and convolution becoming multiplication. speech models read spectrograms (windowed DFTs), and some architectures replace or speed up mixing layers with the FFT.

3. From these labs to the AI labs

Once these mechanisms feel familiar, the AI section's labs assemble them into the parts of a real model: Activation Functions (the nonlinearity between affine layers), Latent Space (meaning as geometry in many dimensions), Attention (dot products, softmax and weighted averages working together) and Gradient Descent (how all the weights are learned).

4. What the labs leave out

Every lab works in two or three dimensions so you can see it. A real model does the same operations in thousands of dimensions, with billions of learned weights, stacked in dozens of layers — and some of what makes it work (tokenization, the training data, scale itself) is not a matter of geometry at all. What carries over unchanged is the mechanism: the angles, projections, weighted sums and derivatives you can drag around here are the same ones running inside the model, just many more of them at once.

Start: Basis Functions →