Theory test: Training deep networks

6 questions. Pass with 5 correct.

1. 01 You initialise every weight of a hidden layer to the same value. What happens?

   Training is fasterAll units in the layer compute and learn the same thing, so the layer acts like one unitThe network can't runNothing; biases break the symmetry
2. 02 He initialisation for ReLU layers draws weights with variance:

   $1$$1/n_{\text{in}}$$2/n_{\text{in}}$$2/(n_{\text{in}}+n_{\text{out}})$
3. 03 Adam maintains, for each parameter:

   Only the latest gradientRunning averages of the gradient and of the squared gradientA separate learning rate tuned by handThe Hessian matrix
4. 04 With inverted dropout and $p=0.5$, what happens during training and at test time?

   Training: drop half the units and double the survivors. Test: use all units unchangedTraining: drop half the units. Test: halve all activationsBoth: drop half the unitsTraining: use all units. Test: drop half
5. 05 Early stopping means:

   Stopping when training loss reaches zeroKeeping the weights from when validation loss was lowestTraining for exactly one epochStopping when the learning rate reaches zero
6. 06 How does momentum help in a long, narrow valley?

   It increases the learning rate everywhereGradient components that flip sign cancel out, while consistent ones accumulate speed along the valleyIt removes the need for gradientsIt moves straight to the global minimum