Protein sequence generation has traditionally relied on autoregressive and diffusion based models, while Fourier-based approaches remain largely unexplored in this domain. We apply FNet, a Fourier transform based architecture originally developed for natural language processing, to protein sequence generation and compare its performance against ESM2’s direct masked sampling strategy and the autoregressive model ProGen. All three models were trained and evaluated on a curated kinase dataset under identical experimental conditions. FNet achieves substantial computational gains over ProGen, reducing generation time by 95% and memory consumption by 39%, while producing sequences with improved amino acid compositional accuracy and competitive structural plausibility scores. ESM2 generates sequences twice as rapidly as FNet, whereas ProGen shows stronger short-range motif preservation. These results indicate that Fourier-based token mixing provides a complementary alternative to attention based and autoregressive generation paradigms for protein sequence modeling, highlighting clear trade-offs between computational efficiency, global sequence statistics, and local motif fidelity. While attention-based models remain superior in overall performance, Fourier-based encoders demonstrate promise as efficient components within future hybrid or large scale protein modeling frameworks, with current limitations largely attributable to FFT implementation overhead rather than architectural constraints. Code available here: https://github.com/bashirgit/Fourier_vs._Attention .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Fourier vs. Attention: A Re-look at Protein Sequence Generation Models

  • Rohann Rahul Shah,
  • Rishabh Kartik Jawagal,
  • Sachin Rajpurohit,
  • Nikhilesh Venkat,
  • Namita Achyuthan,
  • Bhaskarjyoti Das

摘要

Protein sequence generation has traditionally relied on autoregressive and diffusion based models, while Fourier-based approaches remain largely unexplored in this domain. We apply FNet, a Fourier transform based architecture originally developed for natural language processing, to protein sequence generation and compare its performance against ESM2’s direct masked sampling strategy and the autoregressive model ProGen. All three models were trained and evaluated on a curated kinase dataset under identical experimental conditions. FNet achieves substantial computational gains over ProGen, reducing generation time by 95% and memory consumption by 39%, while producing sequences with improved amino acid compositional accuracy and competitive structural plausibility scores. ESM2 generates sequences twice as rapidly as FNet, whereas ProGen shows stronger short-range motif preservation. These results indicate that Fourier-based token mixing provides a complementary alternative to attention based and autoregressive generation paradigms for protein sequence modeling, highlighting clear trade-offs between computational efficiency, global sequence statistics, and local motif fidelity. While attention-based models remain superior in overall performance, Fourier-based encoders demonstrate promise as efficient components within future hybrid or large scale protein modeling frameworks, with current limitations largely attributable to FFT implementation overhead rather than architectural constraints. Code available here: https://github.com/bashirgit/Fourier_vs._Attention .