Cascade Pre-attention: Regulating Neuronal Activation Distributions in MetaFormer-Based Spiking Neural Networks
摘要
Spiking Neural Networks (SNNs), considered third-generation neural networks due to their biological plausibility, possess inherent low-power traits that make them promising for edge computing. However, their lower performance versus traditional Artificial Neural Networks (ANNs) has limited their edge deployment. In this research, we discover that neuronal activation distributions in MetaFormer models—widely recognized as top-performing architectures—differ significantly between SNNs and ANNs, contributing to the performance gap of MetaFormer architectures in SNNs. To address this, we propose a plug-and-play Cascade Pre-Attention mechanism for all MetaFormer-based SNN networks. This mechanism leverages global features from MetaFormer’s convolutional components to compute attention weights, applied to subsequent self-attention blocks to regulate neuronal activation distributions. Experiments show MetaFormer models with Cascade Pre-Attention achieve substantial performance improvements in classification and detection tasks while consuming less energy, with negligible additional parameters. Our approach establishes a foundation for SNN deployment in edge devices and applications.