Logistic regression

Classificationabout 9 minutes

Is this email spam? Is this tumour malignant? Will this customer cancel? Binary classification asks for a yes or no, and ideally for a probability of yes, so that we can decide how cautious to be. Linear regression isn't suitable: its output is unbounded, so it can predict probabilities of 1.7 or −0.3, and a few extreme examples can tilt the whole line.

The sigmoid

Logistic regression keeps the weighted sum $z = w^{⊤} x + b$ (called the logit) and passes it through the sigmoid (logistic) function, which squashes any real number into $(0, 1)$ :

σ (z) = \frac{1}{1 + e ^{- z}}, P (y = 1 ∣ x) = σ (w^{⊤} x + b)

$σ (0) = 0.5$ : a logit of zero means a coin flip.
Large positive $z$ gives probabilities near 1; large negative near 0.
$σ (- z) = 1 - σ (z)$ , and its derivative is neatly $σ (z) (1 - σ (z))$ .

Logistic regression on one feature. The weight sets how steep the transition is; the bias shifts where it happens. Points are coloured by their true class.

What the weights mean: log-odds

Invert the sigmoid and you find the logit is the log-odds of the positive class. So logistic regression is a linear model of log-odds: each unit increase in feature $x_{j}$ multiplies the odds by $e^{w_{j}}$ . Medical papers report exactly these numbers as odds ratios.

lo g \frac{P ( y = 1 ∣ x )}{1 - P ( y = 1 ∣ x )} = w^{⊤} x + b

From probability to decision

To make a hard decision, compare the probability to a threshold, usually 0.5, which is the same as checking the sign of $z$ . But the threshold is a choice: a cancer screen might flag anything above 0.1 because missing a tumour costs far more than a follow-up test. Keeping the probability and choosing the threshold separately is one of the big advantages of probabilistic classifiers. You'll tune thresholds in module 3.

try itedit the code and run it, nothing can break

python sleeps until you press run

Logistic regression is a linear model of the log-odds, turned into a probability by the sigmoid. Thresholding that probability gives a decision, and the threshold is yours to choose.

← Least squares, two ways