Matrices as machines

A matrix is a grid of numbers with $m$ rows and $n$ columns. You can read it two ways, and both matter in machine learning: as a table of data, and as a machine that transforms vectors.

A dataset is a matrix

Stack $n$ feature vectors as rows and you get the design matrix $X$, with shape $(n,d)$: one row per example, one column per feature. The iris dataset is a $150\times 4$ matrix. By convention $X$ has examples in rows; the labels go in a separate vector $\mathbf{y}$ of length $n$.

Matrix times vector

Multiplying a matrix by a vector produces a new vector. Each output element is the dot product of one row of the matrix with the vector:

Now apply that to the design matrix. If $\mathbf{w}$ holds one weight per feature, then $X\mathbf{w}$ is a vector with one number per example: the dot product of each row with the weights. That is a linear model scoring the entire dataset in one operation. No loop over examples, and on a GPU those dot products run in parallel.

Matrices transform space

The second reading: a $2\times 2$ matrix moves every point of the plane. Its first column says where the arrow $(1,0)$ lands; its second column says where $(0,1)$ lands. Everything else follows, because every vector is a mix of those two. These are linear transformations: grid lines stay straight, parallel and evenly spaced, and the origin stays put. Rotations, stretches, shears and reflections are all matrices.

The determinant is the factor by which the transformation scales areas (volumes in higher dimensions). A determinant of 0 means the matrix squashes the plane onto a line: information is destroyed and the transformation cannot be undone, so the matrix has no inverse. That will matter when we solve for regression weights in module 2.

Matrix times matrix

Multiplying two matrices applies one transformation after the other. The rule for shapes is the one you will check a thousand times when debugging neural networks: inner dimensions must match, and they disappear.

• Order matters: $AB\ne BA$ in general. Rotating then stretching is not the same as stretching then rotating.
• The transpose $A^{\top }$ flips rows and columns, turning an $(m,n)$ matrix into $(n,m)$. It shows up whenever we need shapes to line up, for example $X^{\top }X$ is $(d,d)$.
• The identity $I$ has ones on the diagonal and leaves vectors unchanged: $I\mathbf{x}=\mathbf{x}$.

A neural network layer is a matrix multiply followed by a simple non-linear function. Get comfortable with shapes now: (batch, features_in) @ (features_in, features_out) → (batch, features_out).