Self-Attention and Transformers

After feed-forward and recurrent, the transformer architecture is a third kind of network. This chapter dissects the transformer architecture starting from the core concept of attention. It outlines its building blocks with small programs in PyTorch so that the reader can experiment with them. The chapter concludes with an implementation of a sequence annotation and a classifier with the encoder part of a transformer.