A Comparative Study of GPT-2 and GPT-2 Based On Enhanced Self-attention Mechanism

In natural language processing, the quality of language models impacts applications such as machine translation and speech recognition. GPT-2, a powerful auto-regressive model with 150 million parameters, performs exceptionally well in various tasks but struggles with computational efficiency for long sequences. We have developed an optimization strategy to mitigate this issue by randomly shortening the auto-regressive length during generation. Our strategy was tested on the GPT-2 medium model using BLEU as the evaluation metric. The results revealed significant improvements in the BLEU scores, with the optimized model outperforming the original. Furthermore, the optimization also improved scores in both the top and bottom 10% of the data. Despite the promising results, there is still room for further exploration and improvement. We are currently investigating adaptive adjustments to the auto-regressive length and applying this strategy to other models, such as GPT-3. In summary, our research proposes a new strategy that enhances GPT-2’s efficiency and boosts its performance, as evidenced by the improved BLEU scores. This strategy provides valuable insights for future language model optimization, holding the potential to advance the field of NLP.