Gradient Descent
Training is just walking downhill, one step at a time
Every weight in an LLM is nudged the same way, over and over: compute how the loss responds to a small change in that weight, then move a little in the direction that makes it smaller. This lab shows that mechanism on a 2-D loss surface you can actually see — click a point to drop a ball on it, then step or play the descent and watch it follow −∇L(θ) downhill. Push the learning rate too far and watch the same descent that converges smoothly start to oscillate, then blow up.Descent Lab: Following −∇L(θ) Downhill
This tool renders one scalar loss $L(\theta)$ over a 2-D parameter $\theta=(x,y)$ and lets you watch plain gradient descent, with optional momentum, walk downhill on it step by step. The sections below explain the update rule, what the three surfaces are built to demonstrate, and where a 2-D picture like this one stops being an honest stand-in for what training an LLM actually looks like.
1. The update rule
Gradient descent repeats one line: $\theta \leftarrow \theta - \eta \nabla L(\theta)$. The gradient $\nabla L(\theta)$ points in the direction of steepest increase in loss, so stepping against it — scaled by the learning rate $\eta$ — is the locally best move toward lower loss. In a real network $\theta$ is every weight and bias at once, often billions of numbers, and $\nabla L$ is computed all in one pass by backpropagation. Here $\theta$ is just two numbers, small enough to plot, so the same update rule that trains a transformer is visible as a ball rolling downhill.
2. What momentum changes
With momentum $\beta$, the update keeps a running velocity instead of jumping straight along the current gradient: $v \leftarrow \beta v - \eta \nabla L(\theta)$, then $\theta \leftarrow \theta + v$. At $\beta = 0$ this is identical to plain gradient descent — the slider's default. Raise it and the ball carries speed from previous steps, which smooths out oscillation across a steep, narrow valley (try it on the Convex Bowl) and can carry a descent straight through a shallow local dip it would otherwise stall in. It is the same mechanism behind SGD-with-momentum and, loosely, the "memory" term inside Adam.
3. The three surfaces
- Convex Bowl — $L=x^2+3y^2$. One global minimum at the origin, no local minima to get stuck in. Because the bowl is steeper along $y$ than $x$, a learning rate that is safe for $x$ can already be too large for $y$: push $\eta$ past roughly $1/3$ and the same descent that converged smoothly starts overshooting the valley wall every step, then diverges outright. This is the cleanest place to see why a learning rate has an upper bound at all — it is set by the surface's curvature, not by feel.
- Double Well — $L=(x^2-1)^2+y^2$. Two symmetric global minima at $(\pm1,0)$ and a saddle point at the origin. Which minimum a descent lands in depends entirely on which side of $x=0$ it starts on — a direct, visible example of how the same optimizer, same learning rate, same loss function can land in different final solutions depending only on initialization.
- Rugged (LLM-like) — a bowl with a wave folded into it, $L=0.15(x^2+y^2)+0.85\sin(1.8x+0.4)\cos(1.8y)$, producing several local minima of different depths. It is a loose visual analogy for a real training loss landscape, named for that resemblance rather than derived from one — see the honesty limit below for exactly what that analogy does and does not claim.
4. Reading the canvas and the minima markers
The background is $L(x,y)$ itself, sampled on a grid and banded into shells like a topographic map — darker means lower loss, brighter/warmer means higher. The small diamond markers are not hand-placed: for the Double Well they are the two closed-form minima, and for the Rugged surface they come from running short gradient-descent refinements from a grid of seed points and keeping where they settle, deduplicated — the same kind of coarse multi-restart search used in practice when a loss surface has no closed form. When your trajectory converges, the readout compares its final loss against every marker's to say whether you found the global minimum or a shallower local one.
5. Honesty limit: this is a toy, deliberately
A real LLM's loss surface lives in a space with as many dimensions as the model has parameters — commonly billions — and nobody can plot it, walk it, or reason about its curvature by eye the way this demo lets you reason about a 2-D bowl. Two things this tool gets right about the real thing: the update rule itself is exact (this literally is gradient descent, not a cartoon of it), and the qualitative lessons — learning rate has a stability ceiling set by curvature, momentum smooths oscillation, initialization can decide which minimum you reach, non-convex surfaces have local minima and saddle points — do carry over. What does not carry over: production LLM training almost never uses plain SGD with a fixed $\eta$; it uses adaptive optimizers like Adam that rescale each parameter's own step size from running gradient statistics, plus a learning-rate schedule that changes $\eta$ over the course of training. Neither adaptive per-parameter scaling nor scheduling is modeled here — this tool is the single mechanism those methods are built on top of, not a simulation of the methods themselves.