Theory test: Attention and transformers

6 questions. Pass with 5 correct.

1. 01 Why are attention scores divided by $\sqrt{d_k}$?

   To make attention fasterDot products grow with dimension; scaling keeps their variance near 1 so the softmax doesn't saturateTo make the weights sum to 1To hide future tokens
2. 02 What does a causal mask do?

   Removes padding tokensLets each position attend only to itself and earlier positionsRandomly drops attention weightsMakes attention bidirectional
3. 03 The attention score table for a sequence of $n$ tokens costs:

   $O(n)$$O(n\log n)$$O(n^2)$$O(1)$
4. 04 A model has $d_{\text{model}}=512$ and 8 attention heads. Each head works in:

   512 dimensions64 dimensions8 dimensions4,096 dimensions
5. 05 Without any position information, self-attention:

   Works the same on any sentence order: it can't tell word order apartOnly sees the first tokenBecomes an RNNCan't be trained
6. 06 In a pre-norm transformer block, the attention sub-layer computes:

   $\mathbf{x}\leftarrow \mathrm{Attention}(\mathbf{x})$$\mathbf{x}\leftarrow \mathbf{x}+\mathrm{Attention}(\mathrm{LayerNorm}(\mathbf{x}))$$\mathbf{x}\leftarrow \mathrm{LayerNorm}(\mathbf{x})\cdot \mathrm{Attention}(\mathbf{x})$$\mathbf{x}\leftarrow \mathbf{x}-\mathrm{Attention}(\mathbf{x})$