Transformer is a neural network architecture that uses the encoder-decoder framework, but without recurrence or convolution. In transformers, both the encoder and the decoder have multiple transformer blocks—feed-forward stacks/layers (six layers in the original paper, Vaswani et al. 2017). A key feature is that transformers use both attention and self-attention to jointly attend to information from different layers at different input and output positions. Attention was introduced in seq2seq models to allow the decoder to be influenced (guided) by the encoded representation of the preceding input (Bahdanau et al. 2014). In early LSTMs the decoder was influenced by a single vector that represented the whole preceding input, and with the introduction of attention, the decoder could be influenced by several representations from the input. In transformers, both the encoder and the decoder can attend to multiple vectors in different layers of the encoder (and the decoder layers can also attend to layers within the decoder). The attention mechanism allows any block in a transformer to be affected by a very long chunk of context (preceding context, or even subsequent context of the input), much larger than what was possible with LSTM models. In principle, the context can be enlarged by adding more attention units (and hardware processing capacity). The term self-attention relates to attention blocks that are within the same feed-forward module (encoder or decoder), processing the same input sequence. With self-attention, each element (vector) in a sequence of input elements is transformed as a weighted average with the weights of the rest of the sequence. This allows every element to be weighted according to its importance in the input sequence. Self-attention mechanism relates different positions of a single sequence to compute a weighted representation of the same sequence. Moreover, for any layer of input, several attention blocks can attend to it in parallel, with different connections; such an arrangement is termed attention heads. The output vectors of attention heads are concatenated, and the result becomes the input to a next layer of processing. The transformer architecture has become a de-facto standard for neural NLP processing since 2018, being the basis for Large Language Models (LLMs).