Matrices
A matrix is completely determined by where it sends two vectors
M = [î→ | ĵ→] moves every point of the plane at once, but it never moves the origin and it never curves a straight line. Drag the tip of î or ĵ — the two columns of M — and watch the grid, the shaded unit square, and the free vector v follow along exactly. Column 1 is where (1,0) lands; column 2 is where (0,1) lands; every other point is just a weighted mix of the two. Pick a preset to watch it morph in from the identity, or scrub the Flow slider yourself to see the plane move continuously through every matrix in between.Matrix Transformation
M is the identity — it leaves every vector exactly where it is.
Standard Matrix Operations in Large-Scale ANNs
In large-scale Artificial Neural Networks (ANNs)—especially modern Transformer-based LLMs, Diffusion models, and Deep Learning systems—General Matrix Multiplication (GEMM) dominates up to 90%+ of all computational workloads. However, training and running inference on these models requires a specific set of standard linear algebra and tensor operations.
1. General Matrix Multiplication (GEMM) & Batched GEMM
GEMM is the primary computational building block of deep learning hardware (Nvidia Tensor Cores, Google TPUs, Apple Neural Engine).
- Linear / Fully-Connected (FC) Layers: The classic transformation multiplying a batch of input vectors $X$ by a weight matrix $W$: $$Y = XW + B$$
- Batched GEMM (BMM): Used extensively in Multi-Head Attention, where matrix multiplication is performed independently across multiple heads and batch dimensions simultaneously: $$\text{Attention Scores} = Q K^T$$ $$\text{Attention Output} = A \cdot V$$ (Where $Q, K, V$ are the Query, Key, and Value matrices).
2. Matrix Transposition & Tensor Permutation
Matrix transposition flips row and column indices ($(A^T)_{ij} = A_{ji}$), while multi-dimensional tensor permutation reorders axes in memory.
- Backpropagation (Gradient Computation): During training, calculating the gradient with respect to the inputs or weights requires transposing the weight matrix: $$\nabla_X L = (\nabla_Y L) W^T$$
-
Multi-Head Split & Merge: Moving tensors between layout formats like
[Batch, Sequence, Heads, Dimension]and[Batch, Heads, Sequence, Dimension]to prepare data for parallelized attention GEMMs.
3. Outer Products ($\mathbf{u} \otimes \mathbf{v}$)
An outer product takes two vectors of length $m$ and $n$ and produces an $m \times n$ matrix:
$$A = \mathbf{u} \mathbf{v}^T$$- Weight Gradient Calculation: During backpropagation, updating the weight matrix of a linear layer requires an outer product between the input activation vector $\mathbf{x}$ and the incoming loss gradient vector $\boldsymbol{\delta}$: $$\frac{\partial L}{\partial W} = \mathbf{x} \boldsymbol{\delta}^T$$
4. Element-Wise (Hadamard) Operations
These operate individually on corresponding elements of matrices of the same shape ($C = A \odot B$).
- Non-Linear Activations: Applying functions like GELU, SiLU/Swish, or ReLU across weight matrices element-by-element.
- Gated Architectures (e.g., SwiGLU, LSTMs): Multiplying two matrix outputs element-wise to form feature gates: $$\text{SwiGLU}(x) = (\text{SiLU}(x W_1)) \odot (x W_2)$$
- Residual (Skip) Connections: Element-wise addition ($X_{out} = X_{in} + F(X_{in})$) used in almost every layer of Transformers and ResNets to prevent vanishing gradients.
5. Axis Reductions & Normalizations
Operations that aggregate matrix values along specific dimensions (rows or columns) down to vectors or scalars.
- LayerNorm / RMSNorm: Computes the mean and variance across the feature dimension (columns) to normalize inputs before GEMM steps: $$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d} \sum_{i=1}^d x_i^2 + \epsilon}} \odot \gamma$$
- Softmax Normalization: Sums exponentiated values across rows to convert raw attention scores into a valid probability distribution.
- Cross-Entropy Loss: Summing and averaging losses across sequence lengths and batch dimensions.
6. Low-Rank & Sparse Matrix Operations
Because large-scale models suffer from high memory bandwidth bottlenecks, optimized implementations often rely on structured matrix formats:
- Low-Rank Adaptation (LoRA): Replaces a large frozen weight update $\Delta W$ with two smaller, factorized low-rank matrices $A$ and $B$: $$W' = W + (A \cdot B)$$ (Where $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$, with $r \ll d$).
- Sparse-Dense Matrix Multiplication (SpMM / SDDMM): Used in Mixture-of-Experts (MoE) routing, sparse attention kernels (e.g., FlashAttention), and pruned models where large portions of matrices contain zeros.
- Quantized GEMM: Performing matrix operations using low-precision integer or floating-point types (e.g., INT8, INT4, FP8, FP4) and scaling the resulting sub-matrices back to standard precision via FP32 scale factors.