Decision boundaries and many classes

Classificationabout 10 minutes

Decision boundaries

A classifier divides the input space into regions, one per predicted class. The border between regions is the decision boundary. For logistic regression the boundary is where $p = 0.5$ , which is where $w^{⊤} x + b = 0$ : a straight line in 2-D, a plane in 3-D, a hyperplane in general. The weight vector $w$ is perpendicular to it, pointing toward the positive class.

Two features, two classes. Rotate and shift the line. The shading is the predicted probability; only a straight boundary is possible.

A straight boundary is a real limitation. If the positive class forms a ring around the negatives, no line separates them. You can either engineer features (add $x_{1}^{2} + x_{2}^{2}$ and the ring becomes separable) or use a model that learns non-linear boundaries: neighbours and trees in the next unit, neural networks in module 4. A neural network classifier is, in effect, logistic regression on top of features it learns itself.

More than two classes: softmax

With $K$ classes, give each class its own weight vector and compute $K$ logits $z_{k} = w_{k}^{⊤} x + b_{k}$ . The softmax turns them into a probability distribution: exponentiate (making everything positive) and normalise (making them sum to 1):

softmax (z)_{k} = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}

Drag the logits. Softmax preserves their order, exaggerates gaps, and always sums to 1.

With $K = 2$ , softmax reduces exactly to the sigmoid of the difference of the two logits.
Adding the same constant to every logit changes nothing, because it cancels in the ratio. Implementations subtract the maximum logit before exponentiating so that $e^{z}$ never overflows: $e^{1000}$ is infinity in floating point, but $e^{1000 - 1000} = 1$ .
The loss is categorical cross-entropy, $- lo g p_{true class}$ , and its gradient with respect to the logits is again prediction minus truth: $p - y_{one-hot}$ .

Softmax regression (multinomial logistic regression) has decision boundaries that are straight lines between every pair of classes. It is also, exactly, the final layer of nearly every neural network classifier: an image model's last layer is a softmax over 1,000 ImageNet classes, and a language model's is a softmax over its whole vocabulary.

try itedit the code and run it, nothing can break

python sleeps until you press run

Linear classifiers draw straight boundaries. Softmax generalises the sigmoid to $K$ classes and sits at the end of almost every neural network classifier.

← Cross-entropy loss