Theory test: Neurons and networks

6 questions. Pass with 5 correct.

1. 01 Two linear layers stacked with no activation between them are equivalent to:

   A deeper, more powerful modelA single linear layerA ReLU networkA decision tree
2. 02 What are $\mathrm{ReLU}(-2)$ and the derivative of ReLU at $z=3$?

   −2 and 10 and 10 and 02 and 3
3. 03 How many parameters does a dense layer from 784 inputs to 128 units have (with biases)?

   100,352100,480912784
4. 04 Why can't a single perceptron learn XOR?

   XOR needs more training dataThe XOR classes aren't linearly separableThe learning rate is too smallXOR has too many inputs
5. 05 Why do deep stacks of sigmoids train poorly?

   Sigmoids are slow to computeTheir derivative is at most 0.25 and near 0 when saturated, so gradients shrink layer by layerThey can't output negative numbersThey have too many parameters
6. 06 What does the universal approximation theorem guarantee?

   Gradient descent will find a perfect networkA wide enough one-hidden-layer network can approximate any continuous function on a bounded regionDeep networks never overfitOne hidden unit is enough for any function