Matrix Lab: Non-Linear Activation Visualizer

1. Why non-linearity bends the plane

A linear transformation Mx can rotate, scale, shear, or reflect the plane, but it always keeps straight lines straight and the origin fixed — that is what "linear" means. Composing M with an element-wise activation $\varphi$ breaks the straight-line guarantee: $y=\varphi(Mx)$ is free to curve grid lines, clip half the plane to zero, or (with a gate) let a second matrix decide how much of the output survives at each point. That non-linearity is what lets stacks of these layers approximate the complex, curved decision boundaries real networks need.

2. The four activations in the Lab

• Linear (identity): $\varphi(z)=z$. No bending — this is the baseline "M alone" view, equivalent to the existing Matrix Lab.
• ReLU: $\varphi(z)=\max(z,0)$. Every coordinate with a positive pre-activation is left untouched; anything negative is clipped flat onto the axis. The grid visibly snaps into a "hinge" at zero.
• GELU: rendered with the standard tanh approximation, $\varphi(z)\approx 0.5z\left(1+\tanh\left[\sqrt{2/\pi}\,(z+0.044715z^3)\right]\right)$. Unlike ReLU it dips slightly negative before rising, so the grid curves smoothly through the origin instead of creasing.
• SwiGLU (gated): $\text{SwiGLU}(x)=\text{SiLU}(xW_1)\odot(xW_2)$ — the Gate matrix $W_1$ (blue) and Content matrix $W_2$ (gold) each apply their own linear map, then the gate's SiLU output multiplies the content's output element-wise. The $\beta$ slider controls the gate's steepness via $\text{SiLU}(z)=z\cdot\sigma(\beta z)$: low $\beta$ gates softly, high $\beta$ pushes the gate toward a hard ReLU-like switch.

Correction from the original brief: $\beta$ only ever sharpens the SiLU gate inside SwiGLU — the GELU option here uses the fixed tanh approximation above and does not read the $\beta$ slider, since standard GELU has no such parameter. The slider is disabled outside SwiGLU mode to make that boundary visible rather than silently doing nothing.

3. Reading the canvas

Grid lines and the unit square are sampled at many points along each line, mapped through $\varphi(Mx)$, and connected — that dense sampling is what produces the smooth-looking curves; it is a rendering approximation, not literal Bezier-curve fitting. Drag the content matrix's $\hat{\imath}$ or $\hat{\jmath}$ handles (or type into the number grid) to reshape $M$; drag the free vector $v$ to see exactly where any point lands. In SwiGLU mode the gate matrix $W_1$ is numeric-input only — with two matrices acting at once, only one pair of handles can be meaningfully dragged on a single 2-D canvas, so the content matrix (the "underlying linear transformation" per the design brief) is the one exposed to direct manipulation.

The Residual toggle adds the skip connection $X_{out}=X_{in}+F(X_{in})$: it sums the original point back into the warped one, which is why turning it on visibly "un-warps" the grid — the identity path guarantees at least some of the original structure survives underneath the non-linearity, exactly as it does in a Transformer or ResNet block.

4. The gradient heatmap and dead neurons

The background glow is the local gradient magnitude $\lVert J_{\varphi\circ M}(x)\rVert$ — the size of the output's Jacobian at each point, estimated numerically by finite differences rather than a closed-form derivative, so it stays correct across every activation without a separate formula per case. Bright means a small nudge to the input still moves the output — gradient is flowing and the weights feeding that region can keep learning. Dark means the output barely moves — the gradient has vanished.

• ReLU shows a hard blackout across the whole region where every output coordinate has been clipped to zero: the classic "dead ReLU" problem, visualized as a literal void rather than a training-log statistic.
• GELU and SwiGLU keep a continuous, non-zero glow across the origin, since both retain a small negative-side response instead of a hard clip — this is the mechanism papers cite for why smooth gates train more reliably than ReLU in very deep stacks.

Correction from the original brief: the brief's backprop formulas, $\nabla_X L=(\nabla_Y L)W^\top$ and $\partial L/\partial W=x\delta^\top$, are exactly correct for a bare linear layer — and that is where they already appear, worked through in full, on the Matrix Lab's "Matrices & ANNs" tab. Applied to this tool's $\varphi(Mx)$ as written, though, they would be wrong: differentiating through an activation (and, for SwiGLU, through a Hadamard gate) requires an extra factor of the activation's own local slope, $\nabla_X L=\left(\nabla_Y L\odot\varphi'(z)\right)W^\top$, before the transpose trick applies. Rather than restate that formula loosely, this tool computes the real quantity numerically — that is exactly what the heatmap and the "$\lVert\nabla\varphi\rVert$ at v" stat show.

5. One more accuracy note: the origin

The brief states that "non-linear activations and biases" displace the origin. In this tool that is only half true: ReLU, GELU, and SiLU all satisfy $\varphi(0)=0$, so with no explicit bias term (this demo has none — only $W_1$, $W_2$, and the residual), the origin stays exactly fixed under every activation on offer. What moves is everything around it — the grid bends, clips, or gates near the origin, but the origin itself only moves once a bias vector is added, which is why this tool omits one rather than gesture at an effect it can't actually render.

6. Pedagogical flow

1. Start on Linear: confirm the grid stays straight — this is just the existing Matrix Lab.
2. Switch to ReLU: watch the heatmap blackout appear exactly where the grid snaps flat.
3. Switch to GELU: same clipping region, but the heatmap stays lit — the "dead zone" recovers.
4. Switch to SwiGLU and adjust $W_1$ (gate) against $W_2$ (content) to see the gated, non-uniform warp modern LLM feed-forward blocks actually use.