From one-hot to embeddings

Sequence modelsabout 10 minutes

To feed text to a network, first split it into tokens (words or pieces of words, module 6) and give each a number from a vocabulary. The obvious vector for token $i$ is one-hot: a vector the size of the vocabulary with a 1 in position $i$ . With 50,000 tokens that's a 50,000-long vector of zeros with a single 1, and worse, every pair of words is equally different: "cat" is as far from "kitten" as from "carburettor".

Embeddings

An embedding maps each token to a dense vector of, say, 300 learned numbers. It's stored as a matrix $E$ of shape (vocabulary, dimension), and looking up token $i$ means taking row $i$ . That lookup is exactly a one-hot vector times $E$ , so an embedding layer is just a linear layer with a cheap implementation:

e_{i} = x_{one-hot}^{⊤} E = E [i, :]

Because embeddings are trained (by backpropagation, as part of some task), words used in similar ways end up with similar vectors. This rests on the distributional hypothesis, summed up by linguist J. R. Firth in 1957: "You shall know a word by the company it keeps."

word2vec and vector arithmetic

word2vec (Mikolov et al., 2013) trained embeddings cheaply by predicting a word from its neighbours (or neighbours from a word) over billions of words. The vectors had a surprising property: directions encoded relationships. The vector from "man" to "woman" is roughly the same as from "king" to "queen", so $king - man + woman$ lands nearest to "queen". Similar offsets appeared for country→capital and verb tenses.

A hand-drawn 2-D sketch of the idea. Real embeddings have hundreds of dimensions, and analogies hold only approximately.

Similarity is measured with cosine similarity, from module 1: direction matters, length mostly reflects frequency.
Embeddings soak up the biases of their training text. Bolukbasi et al. (2016) showed word2vec completing "man is to computer programmer as woman is to homemaker". Debiasing helps but doesn't fully solve it.
In modern language models the embedding layer is the first layer, trained along with everything else, and each token's vector is then refined by context: "bank" in "river bank" and "bank account" ends up represented differently.

try itedit the code and run it, nothing can break

python sleeps until you press run

An embedding is a learned lookup table from tokens to vectors. Training places words used alike close together, and some relationships become directions.

← Convolution and pooling