Dynamic Activation Function of Spatial-Aware for Visual Recognition
摘要
Normalization layers and activation functions are fundamental components in deep neural networks. This paper challenges the conventional separation of these components by proposing Dynamic Spatial Activation (DSA), which combines the regulation of dynamic activation with spatial context modelling. DSA introduces a learnable spatial condition into the dynamic tanh scaling mechanism, formulated as: \(DSA(\textit{x}) = max(\gamma \odot \tanh (\alpha \cdot \textit{BN}(\textit{x}) + \beta \cdot \textit{T}(\textit{x})) +\delta , \textit{x})\) , where \(\textit{T}(\textit{x})\) captures local spatial patterns through depth-wise convolution, while \(\alpha ,\beta \) dynamically balance linear scaling and spatial attention and \(\delta \) is a learnable bias parameter at channel level, together with the scaling factor \(\gamma \) , constitutes the affine transformation used to adjust the dynamic distribution range of the activation output. Extensive experiments on image classification demonstrate that DSA achieves superior performance in lightweight CNNs: 73.194 top-1 accuracy on MobileNet V2 1.0 (+16.832% over ReLU), and ShuffleNet V2 0.5 (+5.18% over ReLU), and this advantage can easily be generalized to tasks involving object detection and semantic segmentation. Our model is open-sourced at https://github.com/zhoulijun11/DSA .