Basic Language Models

This chapter provides an introduction to basic language models, which calculate the probability of a sentence or predict the next word in a sequence. It first covers statistical language models, specifically the n-gram model, which calculates a word's probability based on the preceding n-1 words and uses data smoothing to address the issue of data sparsity. The chapter then introduces neural network language models, including feedforward and recurrent network-based models, as a solution to the limitations of statistical models, such as their inability to learn long-distance semantic relationships. Finally, the text explores the close relationship between language models and word vectors, explaining how models like Word2Vec and ELMo are trained using a language model approach to generate static and dynamic word vectors, respectively.