Transformers have revolutionized natural language processing (NLP) by surpassing recurrent neural networks (RNNs) in capturing long-range dependencies within sequences. This is primarily due to their self-attention mechanism, which allows the model to attend to all parts of the input simultaneously. However, the complex architecture and numerous hyperparameters of transformers pose challenges in achieving optimal performance. This paper conducts a comprehensive comparison of different transformer architectures and hyperparameters. We explore variations in encoder and decoder design, investigate the impact of hidden layer size, number of attention heads, learning rate, and dropout rate, among others. We evaluate the performance of these configurations on natural language generation (NLG). Our findings shed light on the influence of specific architectural choices and hyperparameter settings on transformer performance, offering valuable insights for researchers and practitioners to optimize transformer models for their specific NLP applications.