Transformer

This chapter introduces the Transformer model, a deep neural network architecture based on Self-Attention, which was first proposed by Google in 2017 in the landmark paper “Attention Is All You Need” (Vaswani et al., NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp 5998–6008), and was initially used for natural language processing (NLP) tasks, especially in machine translation, text classification and language modelling. Thanks to its excellent parallel computing characteristics and long-range dependency modelling capability, Transformer has rapidly overtaken traditional Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) to become the dominant model in the field of NLP.