Fundamentals of Vector-Based Text Representation and Word Embeddings

Text representation is the most crucial task for text mining and forms the base for further processing of the textual data for any text-mining task. It is obvious that the natural language text cannot be processed by machine primarily because of its unstructured representation. In addition to the representation format, capturing the semantics of text is a quite complex problem. For efficient computational processing, especially in context of machine learning, the most appropriate representation is the vector-based representation of text. However, the task is not trivial. The text representation started initially with classical bag of word model for representing the text document as vector of words. Machine learning community used the approach extensively, till the emergence of pre-trained embeddings. These pre-trained embeddings are based on language modeling and provide the text representation for deep-learning based models. The embeddings are able to capture the semantics of text to a certain extent. In this chapter, we present classical vector space model with example from biomedical text. The limitations of the model are also highlighted. This is followed by an overview of language modeling and description of word embeddings, including: Word2vec, Glove, FastText, and Biowordvec.