The linear model

Linear regressionabout 9 minutes

Supervised learning means learning from examples that come with answers. You have inputs $x$ (features) and a target $y$ for each, and you want a function that predicts $y$ for inputs it hasn't seen. When $y$ is a number (a price, a temperature, a petal width) the task is called regression. When it's a category it's classification, the subject of the next unit.

The model

Linear regression predicts with a weighted sum of the features plus a constant, the bias (or intercept):

\overset{y}{^} = w_{1} x_{1} + w_{2} x_{2} + \dots + w_{d} x_{d} + b = w^{⊤} x + b

Each weight says how much the prediction moves when its feature increases by one unit, holding the other features fixed. If a house-price model has weight 12,000 on "number of bedrooms", one extra bedroom adds 12,000 to the predicted price, all else equal. That interpretability is a big reason linear models are still everywhere in science, medicine and economics.

The loss: mean squared error

For each example, the residual $y_{i} - \overset{y}{^}_{i}$ is how far off the prediction is. We need one number that summarises all residuals, and the standard choice is the mean squared error:

MSE (w, b) = \frac{1}{n} i = 1 \sum n (y_{i} - (w^{⊤} x_{i} + b))^{2}

Each square's area is one squared residual. The MSE is their average area. Move the line to make the total as small as you can.

Why squares and not absolute values? Three reasons, in increasing depth:

Squares are smooth, so the loss has a gradient everywhere and gradient descent works well. The absolute value has a kink at zero.
Squares punish large errors much more than small ones, which is often what you want (and sometimes isn't: squared loss is sensitive to outliers).
If you assume the targets are the true line plus Gaussian noise, then maximising the likelihood of the data is exactly minimising the squared error. Least squares is maximum likelihood, the idea from module 1.3.

We often report the root mean squared error (RMSE), which is back in the target's units: an RMSE of 0.2 cm means predictions are typically about 0.2 cm off. Another common summary is $R^{2}$ , the fraction of the target's variance the model explains: 1 is perfect, 0 is no better than always predicting the mean.

R^{2} = 1 - \frac{\sum _{i} ( y _{i} - y ^ _{i} ) ^{2}}{\sum _{i} ( y _{i} - y ˉ ) ^{2}}

try itedit the code and run it, nothing can break

python sleeps until you press run

A linear model is a weighted sum. Training it means choosing the weights that minimise the mean squared error, which is the maximum likelihood answer under Gaussian noise.

← Project: Fit a line by gradient descent