Sequences have variable length and order matters: "dog bites man" isn't "man bites dog". A recurrent neural network (RNN) reads one element at a time and keeps a hidden state , a running summary of everything seen so far. Each step combines the new input with the previous state using the same weights:
Sharing weights across time is the sequence version of a CNN sharing weights across space: the network learns one update rule and applies it at every step, so it handles any length. Outputs can be read at every step (tagging each word, predicting the next character) or only at the end (classifying a review).
Backpropagation through time
To train an RNN, unroll it into a long feed-forward graph, one copy per time step, and run ordinary backprop; this is called backpropagation through time (BPTT). Because the weights are shared, each weight's gradient is the sum of its gradients from every step.
Vanishing and exploding gradients, again
The gradient from step back to step passes through copies of the recurrent step, so it's multiplied times by roughly the same matrix (times tanh derivatives, which are at most 1). If that factor's size is below 1 the gradient shrinks geometrically; above 1 it explodes.
- Exploding gradients are handled with gradient clipping: if the gradient norm exceeds a threshold, rescale it down.
- Vanishing gradients are harder: the network simply can't learn that something 50 steps ago matters. In the sentence "The keys that the man by the old wooden door left on the table are missing", choosing "are" depends on "keys", many words earlier. Plain RNNs struggle with such long-range dependencies (Bengio, Simard & Frasconi, 1994).
An RNN updates one hidden state with shared weights at every step. Training multiplies gradients through every step, so they vanish or explode over long sequences.