Sequence-to-Sequence Architectures: Encoder-Decoders and Decoders

The transformer consisted originally of two parts: the encoder and the decoder. This chapter describes this complete structure and, to make it concrete, illustrates it with a machine-translation program in PyTorch. For sake of feasibility, we replace the word input with characters so that readers can train the model on a simple laptop. It produces, nonetheless, surprising good results given its simplicity. The chapter concludes with a description of the decoder alone and how we can apply it to text generation, dialogue, and question answering.