Pretraining an Encoder: The BERT Language Model

Transformers have the capacity to store a huge quantity of information. This chapter describes how we can pretrain the encoder part of a transformer on very large raw corpora to fit their parameters from word associations. It then describes how to fine-tune the parameters for applications such as classification, sequence annotation, and question answering.