5 hidden checks
Implement the most used optimiser in deep learning and the standard ReLU initialisation.
he_init(n_in, n_out, rng): return weights of shape(n_in, n_out)drawn from a normal distribution with mean 0 and standard deviation , usingrng.normal.adam_step(param, grad, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8): one Adam update as in the lesson, for step numbert(starting at 1). Return the new(param, m, v). Don't modify the inputs in place.
- he_init has the right shape and scalenot run
- he_init draws from the given generatornot run
- the first Adam step moves each parameter by about lrnot run
- adam_step matches a reference over several stepsnot run
- adam_step doesn't modify its inputsnot run