Everything is a point

A network never sees words or pictures, only lists of numbers, and every list of numbers is a point in space. Two numbers define a point on a plane. The language model behind this site represents each piece of text as a list of 896 numbers: a point in a space with 896 directions. That space cannot be pictured, but the rules that move points within it are exactly the rules this station demonstrates on a plane.

The task, seen as geometry

Suppose some points are labelled "yes" and others "no". The simplest possible decision is a single straight cut, with everything on one side "yes" and everything on the other "no". Often no straight cut suffices, because the two kinds of point are intermingled. The essential work of a network is to move the points until a straight cut does suffice, with each layer carrying them further toward that arrangement. For a language model the question is not yes or no but "which token comes next?", and step 7 shows that it is answered in the same way.

Three moves per layer

Every layer transforms space in exactly three ways, always in this order:

• Warp: stretch, rotate, slant or flip the whole plane, keeping every straight line straight.
• Shift: translate every point by the same amount.
• Fold: crease the plane so that some regions land on top of others.

The first two belong to long-established mathematics. The third is what makes a network a network, and the station shows why it cannot be omitted.

Two different times: training and answering

Much writing about networks does not say which of two very different phases it refers to, and explanations often falter on that ambiguity. This station states the phase at each point.

• Answering (formally, inference) occurs every time a model is used. Its numbers are fixed. Text goes in, every layer applies its warp, shift and fold, and a prediction comes out. Every move in this station is something that happens during answering.
• Training occurs before a model is released, and it is slow and expensive. The model processes training text, its predictions are compared with what actually followed, and every number is adjusted slightly so that the prediction would have been better. Training never introduces new kinds of move. It only chooses the numbers within them: how far to warp, how far to shift.

In this station you take both roles. Dragging a slider performs training's task by hand, choosing the numbers; watching the grid move is answering, the application of those numbers. The Training or answering? notes mark the distinction where it matters.

Where the idea comes from

The single straight cut is Rosenblatt's perceptron (1958). Minsky and Papert (1969) proved that a single cut cannot solve some very simple problems, and that result shaped the field for more than a decade. The remedy, layers that move the points before the cut is made, became practical with the spread of backpropagation in the 1980s and is the basis of every network in use today.

The claim under test

One warp, one shift and one fold suffice to take three clusters in a row (blue, orange, blue), which no straight line can separate, and make them separable by a single straight cut. Step 6 asks you to do it by hand and then shows what that straight cut looks like on the original plane. It is no longer straight.

The route

2. The warp: drag two arrows, and the whole grid follows.
3. The shift: move the origin, and follow one point's arithmetic as it lands.
4. Stack them: whether chaining warps and shifts can ever bend anything.
5. The fold: several kinds of fold, and how each creases the plane.
6. Origami: the three moves combined to separate the clusters.
7. The real model: the language model behind this site doing the same thing, live.
8. Name it: the standard terminology and the one-line formula.
9. Check yourself: four short questions.

How to use this station

Each step offers something to drag, play or try. Experiment first, then read the notes beneath, which interpret what the panel showed. Hovering over any button describes what it does. Everything runs in the browser except step 7, which sends the text you type to a live model on our server. The strip above records which steps you have visited, and the steps may be taken in any order.

A warp is determined entirely by where it sends two arrows. Every point on this grid is a pair of numbers (x, y), and two arrows serve as the reference: î is the point (1, 0) and ĵ is (0, 1). Drag their tips, and the whole grid follows.

What the panel shows

The two columns of the matrix are exactly where î and ĵ landed. That is all a matrix is: a record of where the reference arrows go. Everything else follows from it.

It is also why the grid never bends. The point (x, y) always lands at x copies of î plus y copies of ĵ. When î moves, every point moves by its own x copies of that change, so straight lines remain straight and parallel lines remain parallel.

Two cases are worth trying. Point î and ĵ in the same direction, and the area falls to 0 as the plane collapses onto a line. Exchange them, and the area becomes negative: the grid has been flipped. Throughout, the origin never moves.

Why "warp" and not "shear"?

A shear is one particular warp: it slides one direction sideways and turns squares into slanted parallelograms, as the Shear button shows. A matrix can also stretch, rotate, flip or collapse the plane, or combine those, and the whole family needs one name. "Warp" is the term used here, in the same sense as in image processing, where OpenCV's warpAffine is this operation plus the shift introduced in the next step.

The family is simpler than it looks: no warp is more complicated than rotate, stretch along two perpendicular directions, rotate again, with a flip if the area comes out negative. Even a shear decomposes this way. Every warp in this step keeps straight lines straight; bending is a different move, and it arrives in step 5.

Training or answering? Both, in different ways. Where î and ĵ point is what training chooses; in a real model those are numbers stored in the network. Moving every point of the grid with them is answering: once the arrows are set, the same warp is applied to every input, every time.

A warp can never move the origin, since 0 copies of î plus 0 copies of ĵ is still 0. A second ingredient removes that limitation: a vector b added to every point, which slides the whole warped grid. Drag the yellow origin to shift and the arrow tips to warp. The warp from step 2 carries over.

What the panel shows

That sum, so many copies of î plus so many copies of ĵ plus the shift, is the entire computation inside a neural-network layer apart from the fold introduced in step 5. Real layers differ only in using longer lists. In Qwen2.5-0.5B, the model behind this site's live labs, each token is a list of 896 numbers, so W has 896 columns instead of 2. The principle is unchanged.

A warp followed by a shift is called an affine map (a linear map plus a shift). It is the term used throughout the literature, and step 8 states it formally.

Training or answering? As with the warp: training chooses b, and answering adds it to every input.

A real network is not one layer but a stack of them, each passing its points to the next. Before adding anything new, it is natural to ask whether enough warps and shifts in sequence could bend the grid into any shape required. Below, a small square of grid passes through two warp-and-shift maps in turn. Follow it from left to right with one question in mind: does any line ever curve?

What the panels show

No grid line curves, whatever pair is tried. The first map keeps lines straight, so the second receives straight lines and keeps them straight in turn. Panels 3 and 4 always end identical: two maps in sequence equal one single map, whose numbers appear in the third box. One warp and one shift do the work of both.

Why this matters

A hundred warp-and-shift maps in sequence still amount to one. A hundred-layer network built only from them could separate two groups of points only with a single straight line, and would be no more capable than one layer. Something must break the straightness, and that is the role of the fold.

Training or answering? Neither, strictly: this is a property of the maps themselves. It explains why training a deep stack of maps without folds would gain nothing.

The most common fold is also the simplest: max(0, v), applied to each coordinate separately. Negative numbers become 0 and positive numbers pass through unchanged. It has a standard name, given in step 8; here it is simply the fold below. Colors show the quadrant in which each dot started.

What the panel shows

With max(0, v), both coordinates of every bottom-left dot become 0, so the whole quadrant lands on (0, 0) and its counter falls to 1 / 36. The top-left and bottom-right quadrants are flattened onto the axes, and only the top-right is left untouched. The fold discards information, and that proves to be exactly what makes it useful.

The other folds

|v| mirrors instead of flattening: the bottom-left lands on the top-right, so nothing is lost, but two quadrants now overlap. tanh squeezes instead of folding: lines curve, but no two points merge. v changes nothing. And 0.5v + 1 is itself a warp-and-shift, so the grid remains a grid: a fold that is a straight line is no fold at all.

Training or answering? Answering. The fold has no numbers of its own for training to choose; it is fixed when the network is designed.

One layer can make a straight cut possible where none existed. An orange group lies between two blue groups, and no straight line can place all the orange on one side and all the blue on the other: the best line classifies little more than two-thirds of the points correctly, as the score below records. The task is to send the points through one layer, a warp, a shift and a fold, and then look for a straight cut. The guided moves below divide the task into four small goals, and Step pauses after each part of the layer to show its effect.

The origami view

Once the cut succeeds, consider the left panel. The cut is straight after the fold, but drawn back onto the original plane it is bent, just as a straight cut through folded paper becomes a bent line when the paper is opened. Each fold contributes a bend. With more layers the bends multiply, which is how a network can draw almost any boundary.

Rotation alone often suffices here: many angles reach 100% without any shift. That works because the orange group sits at the origin, so a rotation sends the two blue groups out along different axes while the orange remains bunched in the corner. With the data off-center, the shift would usually be needed as well. Real models obtain much of that effect by other means, and most of their warps have no shift at all; the notes at the end of step 7 show what a layer actually contains.

Training or answering? Moving the sliders until the score reached 100% was training, performed by hand: a search for good numbers. Pressing Play or Step with the sliders left alone is answering: the numbers stay fixed and the points are carried through the layer. A real network has hundreds of millions of such numbers, and no one chooses them by hand. Training finds them automatically, by adjusting every number in whichever direction improves the score, and it is the subject of its own station.

Moving points until one straight cut separates them is the whole task of a network, and a language model performs the same task at a vastly larger scale. The table sets the model behind this site beside the Origami exercise:

What the panels show

For most of the 24 layers, the point lies nowhere near the correct side of the correct cut. The layers continue to warp, shift and fold it, and only near the end does it arrive where a straight cut can select the answer. This is the Origami exercise's 71% → 100%, carried out in 896 dimensions instead of 2.

Training or answering? Everything in this step is answering. Pressing Run changes none of the model's half-billion numbers; they were fixed when its training ended. The same text run twice produces exactly the same chart. The model has learned nothing from the visitor; it has only carried the point through 24 layers it already had.

Limitations of this view

• The fold is not exactly max(0, v). This model uses a smoother, gated fold called SwiGLU (Shazeer, 2020). The principle is the same; the crease is softer.
• Attention is the other half of each layer. Each layer also draws in information from the other tokens, which is how " France" comes to influence the prediction made at " is". Attention has its own station later in the sequence.
• The layer-by-layer view is approximate. It applies the final cuts to points that are still in motion, a use the cuts were never trained for, which partly explains why the early layers look like noise. The technique is called the logit lens (nostalgebraist, 2020).

The Interpretability lab provides the full version: the lens at every token of the prompt, and what attention is doing.

Every idea on this page has already appeared in the steps above. The table gives the standard names, which open the research literature, and the symbols, which are shorthand for the same operations.

What this does not yet explain

• Who chooses W and b? In the Origami exercise, you set them by hand: six numbers in all. Real networks have millions or billions, found by training. Every formula in the table runs during answering; training is a separate process that runs them, measures the error, and adjusts $W$ and $\mathbf{b}$ (Gradient Descent lab).
• How many folds does a shape need? With enough layers and units, a network can approximate almost any boundary. This is the universal approximation theorem (Cybenko, 1989; Hornik, 1991), and it is the subject of the next station.
• What about 896 dimensions? Everything here works identically with longer lists, but intuition about high-dimensional space requires a station of its own.

Go deeper

• Matrices lab: more warps, and what each one keeps.
• Affine Transforms lab: why the shift fits inside one bigger matrix, and the cost of applying it to a million points.
• Activation Functions lab: ReLU's relatives and why the choice matters.

Four questions on the station's central ideas. Each should be answerable without referring back.