The history of CNNs is short and dramatic, and each landmark added one idea that is still used today.
| year | network | idea |
|---|---|---|
| 1998 | LeNet-5 (LeCun et al.) | conv → pool → conv → pool → dense, trained by backprop; read cheques for US banks |
| 2012 | AlexNet (Krizhevsky, Sutskever, Hinton) | much bigger, ReLU, dropout, data augmentation, trained on two GPUs |
| 2014 | VGG (Simonyan & Zisserman) | uniform stacks of 3×3 convolutions, 16–19 layers |
| 2014 | GoogLeNet (Szegedy et al.) | parallel branches of different kernel sizes (Inception modules) |
| 2015 | ResNet (He et al.) | residual connections; 152 layers |
The ImageNet moment
ImageNet (Deng et al., 2009) is a dataset of over a million labelled photos; its annual challenge (ILSVRC) asked for the correct class among 1,000 in a model's top five guesses. In 2012 AlexNet's top-5 error was 15.3%, against 26.2% for the next-best entry, a margin so large it convinced the field almost overnight. Errors then fell year after year.
Residual connections
By 2015, simply stacking more layers made networks worse, even on training data: a 56-layer plain network trained to a higher error than a 20-layer one. That's not overfitting; the deep one was harder to optimise. ResNet's fix is disarmingly simple: let each block compute a correction to its input rather than a whole new representation.
- If a block isn't useful, it can learn and pass its input through unchanged, so extra depth can't easily hurt.
- The identity path gives gradients a direct route back through every block: , never vanishing just because 's derivative is small.
- ResNets with batch normalisation trained 152 layers and won ILSVRC 2015 with 3.57% top-5 error. Residual connections are now in nearly every deep architecture, including every transformer.
Transfer learning
The features a CNN learns on ImageNet (edges, textures, parts) are useful for almost any vision task. So the standard recipe is to take a pretrained network, replace its final layer, and fine-tune on your own, much smaller dataset. A few hundred labelled images can then be enough. This reuse of pretrained models became the template for language models too. Since 2020, Vision Transformers (Dosovitskiy et al.) apply the transformer of unit 5.3 to image patches and rival CNNs when trained on enough data.
Deep CNNs won ImageNet in 2012; residual connections () made very deep networks trainable; pretrained networks transfer to new tasks.