Pre-trained Large Models

This chapter provides an overview of pre-trained large models, which have established a dominant “pre-training + fine-tuning” paradigm in natural language processing. It examines two main categories: autoregressive language models like GPT, which utilize a Transformer decoder structure to predict the next word, and autoencoder models such as BERT, which use a Transformer encoder and are trained with Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks. The text also introduces knowledge-enhanced models like ERNIE, which improves upon BERT by implementing phrase-level and entity-level knowledge masking strategies. The chapter culminates in a discussion of large language models (LLMs), noting their massive parameter scale and emergent abilities. It concludes by outlining the modern three-stage process for building advanced models like ChatGPT: training a foundational large model, followed by instruction tuning and reinforcement learning from human feedback (RLHF).