Back to Study Material Deep Learning

Word embeddings

Word embeddings are dense, low-dimensional vector representations of words that capture semantic and syntactic relationships. Unlike one-hot encodings, whi...

Word embeddings

Representing Words as Dense Vectors

Word embeddings are dense, low-dimensional vector representations of words that capture semantic and syntactic relationships. Unlike one-hot encodings, which represent each word as a sparse vector with a single 1 and thousands of zeros, embeddings place semantically similar words close together in a continuous vector space, encoding meaning in a form neural networks can process efficiently.

The most influential early method, Word2Vec, trains a shallow neural network on a large corpus using two tasks: predicting a word from its context (CBOW) or predicting context words from a target word (Skip-gram). The weights of the hidden layer become the word embeddings. Words appearing in similar contexts end up with similar vectors, producing a geometry that encodes analogy: the vector for king minus the vector for man plus the vector for woman is close to the vector for queen.

GloVe (Global Vectors) learns embeddings from word co-occurrence statistics across the entire corpus, capturing global as well as local context. Both Word2Vec and GloVe produce static embeddings: a word has a single fixed vector regardless of context, which is a limitation for words with multiple meanings.

Contextual embeddings from models like BERT and GPT solve this by producing a different vector for each occurrence of a word depending on its surrounding context. The word bank has a different embedding when referring to a financial institution versus a river bank.

A concrete example: the Word2Vec embedding for the word doctor has high cosine similarity to nurse, hospital, and patient, reflecting the semantic neighbourhood learned from medical text. Feeding these dense embeddings to downstream models like text classifiers or machine translation systems dramatically outperforms one-hot representations.

Exam tip: interviewers frequently ask why dense embeddings are better than one-hot vectors. Three reasons are standard: they are compact (300 dimensions instead of 100,000), they encode similarity (similar words have similar vectors), and they enable generalisation to unseen word combinations. Knowing these three points and being able to explain each concisely covers most interview depth requirements.