The Long Short-Term Memory network (Hochreiter & Schmidhuber, 1997) attacks vanishing gradients by changing how memory is updated. Alongside the hidden state it keeps a cell state , a conveyor belt that information can ride along almost unchanged. Three gates, each a sigmoid layer outputting numbers between 0 and 1, control it:
- The forget gate decides what to keep from the old cell state (1 = keep everything).
- The input gate decides how much of the new candidate to write.
- The output gate decides what part of the cell to expose as .
- Each is a dense layer of with its own weights.
The key is the additive update of . When the forget gate is near 1, , so gradients flow back through many steps without shrinking, the same trick as a residual connection. The GRU (Cho et al., 2014) merges the cell and hidden state and uses two gates; it performs similarly with fewer parameters. LSTMs powered speech recognition, translation and handwriting recognition through the 2010s.
Sequence to sequence, and its bottleneck
For translation, Sutskever, Vinyals and Le (2014) used two LSTMs: an encoder reads the source sentence into its final state, and a decoder generates the translation from that state, one word at a time. It worked remarkably well, with one obvious weakness: the whole sentence, however long, must squeeze through one fixed-size vector. Performance dropped for long sentences.
Bahdanau, Cho and Bengio (2014) removed the bottleneck. Instead of only the final state, let the decoder look back at all the encoder's states at every step, and learn how much to weight each one: when producing "chat", attend mostly to "cat". That mechanism is attention. In 2017 it turned out you could drop the recurrence altogether and keep only attention, which is the next unit.
LSTMs gate an additive memory so gradients survive many steps. Encoder-decoder RNNs had a fixed-size bottleneck, and attention was invented to look past it.