Distributed Representation

This chapter introduces distributed representation as a foundational technique in natural language processing that maps text into low-dimensional, dense vectors to capture semantic meaning, improving upon traditional one-hot encoding methods that fail to represent relationships between words. The text first covers word-level representations, differentiating between static word vectors, such as Word2Vec and GloVe, which have fixed representations, and dynamic word vectors, like ELMo, that adapt to context to address polysemy. It then progresses to the representation of phrases, discussing compositional methods like the bag-of-words model and more advanced techniques that consider word order. Finally, the chapter addresses sentence representation, outlining universal and task-related methods that leverage recurrent neural networks and pre-trained language models like BERT to encode the meaning of entire sentences.