Theory test: Backpropagation

6 questions. Pass with 5 correct.

1. 01 A variable feeds into two different nodes of a computation graph. Its gradient is:

   The gradient from the first nodeThe sum of the gradients from both nodesThe product of both gradientsZero
2. 02 In $Y=XW$ with $X$ of shape $(32,100)$ and $W$ of shape $(100,10)$, the gradient $\bar{W}$ is:

   $\bar{Y}{X}^{\top }$, shape $(10,32)$$X^{\top }\bar{Y}$, shape $(100,10)$$\bar{Y}{W}^{\top }$, shape $(32,100)$$X\bar{Y}$, shape undefined
3. 03 Why is reverse-mode autodiff used to train neural networks?

   It's more accurate than forward modeOne backward pass gives the gradient of the scalar loss with respect to all parametersIt uses less memoryIt doesn't need the chain rule
4. 04 In the backward pass of ReLU, the incoming gradient is:

   Always passed through unchangedPassed through where the input was positive, zeroed elsewhereMultiplied by the inputSquared
5. 05 For softmax followed by cross-entropy with one-hot targets, $\partial L/\partial \mathbf{z}$ for one example is:

   $\mathbf{p}(1-\mathbf{p})$$\mathbf{p}-\mathbf{y}$$-\log \mathbf{p}$$\mathbf{y}/\mathbf{p}$
6. 06 What is gradient checking?

   Clipping gradients that are too largeComparing analytic gradients against finite-difference estimatesChecking gradients are positiveTesting on the validation set