Spiking Neural Networks (SNNs), considered third-generation neural networks due to their biological plausibility, possess inherent low-power traits that make them promising for edge computing. However, their lower performance versus traditional Artificial Neural Networks (ANNs) has limited their edge deployment. In this research, we discover that neuronal activation distributions in MetaFormer models—widely recognized as top-performing architectures—differ significantly between SNNs and ANNs, contributing to the performance gap of MetaFormer architectures in SNNs. To address this, we propose a plug-and-play Cascade Pre-Attention mechanism for all MetaFormer-based SNN networks. This mechanism leverages global features from MetaFormer’s convolutional components to compute attention weights, applied to subsequent self-attention blocks to regulate neuronal activation distributions. Experiments show MetaFormer models with Cascade Pre-Attention achieve substantial performance improvements in classification and detection tasks while consuming less energy, with negligible additional parameters. Our approach establishes a foundation for SNN deployment in edge devices and applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Cascade Pre-attention: Regulating Neuronal Activation Distributions in MetaFormer-Based Spiking Neural Networks

  • Yukun Xue,
  • Lichen Feng

摘要

Spiking Neural Networks (SNNs), considered third-generation neural networks due to their biological plausibility, possess inherent low-power traits that make them promising for edge computing. However, their lower performance versus traditional Artificial Neural Networks (ANNs) has limited their edge deployment. In this research, we discover that neuronal activation distributions in MetaFormer models—widely recognized as top-performing architectures—differ significantly between SNNs and ANNs, contributing to the performance gap of MetaFormer architectures in SNNs. To address this, we propose a plug-and-play Cascade Pre-Attention mechanism for all MetaFormer-based SNN networks. This mechanism leverages global features from MetaFormer’s convolutional components to compute attention weights, applied to subsequent self-attention blocks to regulate neuronal activation distributions. Experiments show MetaFormer models with Cascade Pre-Attention achieve substantial performance improvements in classification and detection tasks while consuming less energy, with negligible additional parameters. Our approach establishes a foundation for SNN deployment in edge devices and applications.