Scholarlink

<p>Quadratic attention enhances interaction capacity in Transformer models but leads to rapid growth in computational demands as attention rank increases. This paper presents a bounded-rank quadratic attention mechanism where fixed-dimensional feature encodings determine the interaction space and enforce a strict upper bound on attention rank. A Fourier-domain formulation offers a spectral interpretation of the quadratic kernel via unitary transformation while maintaining exact attention computation. The proposed approach achieves bounded-rank attention with computational complexity that scales linearly with sequence length when the encoding dimension remains constant. Experimental validation on a real-world <i>Tea</i> pest image dataset yields 84.5% classification accuracy, surpassing the 82.1% achieved by a standard Vision Transformer under equivalent experimental conditions. Attention processing time per image declines from 14.8 ms to 5.6 ms. Peak memory usage declines from 820 MB to 430 MB. Although memory bandwidth and kernel launch overhead restrict the measured speedup to <InlineEquation ID="IEq1"><EquationSource Format="TEX">\(2.6\times\)</EquationSource></InlineEquation>, the accuracy loss under additive noise reaches 2.9% compared to 3.1% for the baseline, demonstrating comparable robustness. These findings indicate that explicit rank control in attention mechanisms can be realized through representational design, offering an efficient bounded-rank alternative to conventional full-rank attention.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Quantum-inspired quadratic attention with fourier-domain rank control in transformer architectures

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月8日

查看全文
  • MD Tausif Mallick,
  • Saptarshi Banerjee,
  • Ebrahim Abdulla Mattar,
  • Pronaya Bhattacharya,
  • Sujan Pal,
  • Avnish Kumar,
  • Jon Turdiev,
  • Himadri Nath Saha,
  • Amlan Chakrabarti

摘要

Quadratic attention enhances interaction capacity in Transformer models but leads to rapid growth in computational demands as attention rank increases. This paper presents a bounded-rank quadratic attention mechanism where fixed-dimensional feature encodings determine the interaction space and enforce a strict upper bound on attention rank. A Fourier-domain formulation offers a spectral interpretation of the quadratic kernel via unitary transformation while maintaining exact attention computation. The proposed approach achieves bounded-rank attention with computational complexity that scales linearly with sequence length when the encoding dimension remains constant. Experimental validation on a real-world Tea pest image dataset yields 84.5% classification accuracy, surpassing the 82.1% achieved by a standard Vision Transformer under equivalent experimental conditions. Attention processing time per image declines from 14.8 ms to 5.6 ms. Peak memory usage declines from 820 MB to 430 MB. Although memory bandwidth and kernel launch overhead restrict the measured speedup to \(2.6\times\), the accuracy loss under additive noise reaches 2.9% compared to 3.1% for the baseline, demonstrating comparable robustness. These findings indicate that explicit rank control in attention mechanisms can be realized through representational design, offering an efficient bounded-rank alternative to conventional full-rank attention.