Normalization layers and activation functions are fundamental components in deep neural networks. This paper challenges the conventional separation of these components by proposing Dynamic Spatial Activation (DSA), which combines the regulation of dynamic activation with spatial context modelling. DSA introduces a learnable spatial condition into the dynamic tanh scaling mechanism, formulated as: \(DSA(\textit{x}) = max(\gamma \odot \tanh (\alpha \cdot \textit{BN}(\textit{x}) + \beta \cdot \textit{T}(\textit{x})) +\delta , \textit{x})\) , where \(\textit{T}(\textit{x})\) captures local spatial patterns through depth-wise convolution, while \(\alpha ,\beta \) dynamically balance linear scaling and spatial attention and \(\delta \) is a learnable bias parameter at channel level, together with the scaling factor \(\gamma \) , constitutes the affine transformation used to adjust the dynamic distribution range of the activation output. Extensive experiments on image classification demonstrate that DSA achieves superior performance in lightweight CNNs: 73.194 top-1 accuracy on MobileNet V2 1.0 (+16.832% over ReLU), and ShuffleNet V2 0.5 (+5.18% over ReLU), and this advantage can easily be generalized to tasks involving object detection and semantic segmentation. Our model is open-sourced at https://github.com/zhoulijun11/DSA .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Activation Function of Spatial-Aware for Visual Recognition

  • Lijun Zhou,
  • Yunfei Liu,
  • Junran Zhang

摘要

Normalization layers and activation functions are fundamental components in deep neural networks. This paper challenges the conventional separation of these components by proposing Dynamic Spatial Activation (DSA), which combines the regulation of dynamic activation with spatial context modelling. DSA introduces a learnable spatial condition into the dynamic tanh scaling mechanism, formulated as: \(DSA(\textit{x}) = max(\gamma \odot \tanh (\alpha \cdot \textit{BN}(\textit{x}) + \beta \cdot \textit{T}(\textit{x})) +\delta , \textit{x})\) , where \(\textit{T}(\textit{x})\) captures local spatial patterns through depth-wise convolution, while \(\alpha ,\beta \) dynamically balance linear scaling and spatial attention and \(\delta \) is a learnable bias parameter at channel level, together with the scaling factor \(\gamma \) , constitutes the affine transformation used to adjust the dynamic distribution range of the activation output. Extensive experiments on image classification demonstrate that DSA achieves superior performance in lightweight CNNs: 73.194 top-1 accuracy on MobileNet V2 1.0 (+16.832% over ReLU), and ShuffleNet V2 0.5 (+5.18% over ReLU), and this advantage can easily be generalized to tasks involving object detection and semantic segmentation. Our model is open-sourced at https://github.com/zhoulijun11/DSA .