1

One of the tasks can be done with a word2vec model is to find most similar words for a give word using cosine similarity. What is the intuitive explanation for why similar words under a good word2vec model will be close to each other in the space?

1 Answer 1

2

The model is essentially performing compression.

It has to force a large number of distinct words (such as 100,000 or millions) into vectors of a much smaller number of dimensions (such as 100 or 300).

Necessarily, then, a simple "one-hot" encoding, where every word lights up one dimension entirely (1.0), and others not at all (0.0) is not possible. The words must use mixes of values across the small number of dimensions.

In order for such a model to become better on its training task – predicting nearby words – words that often have similar neighbors will grow to have similar coordinates. That allows a 'neighborhood' of the coordinate space to generate similar predictions.

But also, the remaining ways in which those words should predict different neighbors will also tend to be reflected in gradual coordinate changes (directions or neighborhoods).

For example, consider the case of polysemy. The words 'vault' & 'bank' will both appear around 'money' (& related concepts) - but 'bank' in another sense will also appear around 'river' & 'water' (& related concepts). So training in many contexts nudges 'bank' & 'vault' to be more alike (to generate shared 'money'-related predictions). But then other interleaved example usage contexts then nudge 'bank' towards 'river'/'water'.

The net effect of these various tug-of-war backpropagation updates, by the end of sufficient & well-parameterized training, tends to arrange the words in interesting ways - where similar words are nearer, and even certain directions/neighborhoods are vaguely associated with human-describable concepts. (For example, "that direction" might vaguely shift concepts towards money, "this other direction" in terms of gendered concepts, "this 3rd direction" in terms of colors, etc. - though such directions are necessarily fuzzy, imbued by patterns in the training data, can vary from run-to-run, and are not neatly aligned with each dimensions' perpendicula axes.)

Most generally: words that must, for model-optimization purposes, predict the same neighbor words tend to acqire similar coordinates. Their remaining differences are correlated with which neighboring words they need to predict diferently.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.