Scholarlink

Transformer-based language models achieve state-of-the-art performance on many natural language processing tasks. However, such language models have grown very large, precluding their use to many NLP practitioners. In this work, simpler language models based on feed-forward and recurrent neural networks are explored and compared with transformer-based models on a custom benchmark. Results show that although transformer-based models outperform simpler models, their performance gain can be quite small and at the cost of a very strong regularisation.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Are Transformers Always Necessary for Statistical Language Modelling?

  • 章节

  • 发表日期：

    2026年

查看全文
  • Vincenzo Capone,
  • Angelo Casolaro,
  • Gennaro Iannuzzo,
  • Francesco Camastra

摘要

Transformer-based language models achieve state-of-the-art performance on many natural language processing tasks. However, such language models have grown very large, precluding their use to many NLP practitioners. In this work, simpler language models based on feed-forward and recurrent neural networks are explored and compared with transformer-based models on a custom benchmark. Results show that although transformer-based models outperform simpler models, their performance gain can be quite small and at the cost of a very strong regularisation.