Scholarlink

<p>Long-sequence modeling faces critical challenges with the standard Transformer, including quadratic computational complexity and limitations in capturing long-range dependencies and directional token relationships. To address these, we propose Transformer++, a novel framework integrating three core innovations: (1) a hierarchical dual attention mechanism that segments sequences to slash computational costs; (2) direction-aware positional encoding that explicitly models asymmetric token interactions, overcoming the symmetry limitation of standard encodings; and (3) multi-strategy token sampling to select representative tokens for efficient attention computation. Extensive experiments on Baidu Baike, Wikipedia, and the Long Range Arena benchmark demonstrate that Transformer + + significantly outperforms state-of-the-art methods, achieving accuracy gains up to + 4.87% on 12k-token sequences while enabling 2.4× faster inference. This work provides an efficient, scalable solution for demanding long-sequence applications like legal document analysis and medical record modeling.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Transformer++: a long sequence modeling method based on direction-aware dual attention and multi-head sampling

  • 期刊论文

  • 发表日期：

    2025年11月19日

查看全文
  • Ruiqin Wang,
  • Qishun Ji,
  • Zhenzhen Sheng,
  • Yang Qi

摘要

Long-sequence modeling faces critical challenges with the standard Transformer, including quadratic computational complexity and limitations in capturing long-range dependencies and directional token relationships. To address these, we propose Transformer++, a novel framework integrating three core innovations: (1) a hierarchical dual attention mechanism that segments sequences to slash computational costs; (2) direction-aware positional encoding that explicitly models asymmetric token interactions, overcoming the symmetry limitation of standard encodings; and (3) multi-strategy token sampling to select representative tokens for efficient attention computation. Extensive experiments on Baidu Baike, Wikipedia, and the Long Range Arena benchmark demonstrate that Transformer + + significantly outperforms state-of-the-art methods, achieving accuracy gains up to + 4.87% on 12k-token sequences while enabling 2.4× faster inference. This work provides an efficient, scalable solution for demanding long-sequence applications like legal document analysis and medical record modeling.