The selection of activation functions in visual recognition significantly influences training dynamics and task performance. This study introduces an activation function called local spatial and global context activation (SCeLU), which is a conceptually effective activation function. SCeLU extends the Rectified Linear Unit (ReLU) and FReLU to a 3D activation by incorporating a negligible overhead of spatial context conditions. The forms of ReLU and FReLU are \(f\left( x \right) = max\left( {x,0} \right) \) and \(f\left( x \right) = max\left( {x,\textrm{T}\left( x \right) } \right) \) , respectively, where \(\textrm{T}\left( \cdot \right) \) represents the 2D spatial condition. However, SCeLU takes the form of \(f\left( x \right) = max\left( {x,\varPi (x) \cdot \varGamma \left( x \right) } \right) \) , where \(\varPi \left( \cdot \right) \) represents the 3D global context condition and \(\varGamma \left( \cdot \right) \) represents the 2D local spatial condition. Intuitively, the context condition facilitates the modeling of global information, while the spatial condition enhances the capacity for local pixel-wise modeling. By appropriately combining spatial and context conditions, SCeLU demonstrates adaptability to complex visual layouts in various image recognition tasks. By simply changing the activation function, experiments conducted on ImageNet demonstrate a significant enhancement and robustness of SCeLU, particularly for small models, and some enhancements under partially highly optimized large models. Furthermore, our novel SCeLU seamlessly extends to object detection and semantic segmentation tasks, underscoring its effectiveness as an effective alternative in various visual recognition tasks. Our model is open-sourced at https://github.com/YunDuanFei/SCeLU .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Learning Local Spatial and Global Context Activation for Visual Recognition

  • Yunfei Liu,
  • Lijun Zhou,
  • Junran Zhang

摘要

The selection of activation functions in visual recognition significantly influences training dynamics and task performance. This study introduces an activation function called local spatial and global context activation (SCeLU), which is a conceptually effective activation function. SCeLU extends the Rectified Linear Unit (ReLU) and FReLU to a 3D activation by incorporating a negligible overhead of spatial context conditions. The forms of ReLU and FReLU are \(f\left( x \right) = max\left( {x,0} \right) \) and \(f\left( x \right) = max\left( {x,\textrm{T}\left( x \right) } \right) \) , respectively, where \(\textrm{T}\left( \cdot \right) \) represents the 2D spatial condition. However, SCeLU takes the form of \(f\left( x \right) = max\left( {x,\varPi (x) \cdot \varGamma \left( x \right) } \right) \) , where \(\varPi \left( \cdot \right) \) represents the 3D global context condition and \(\varGamma \left( \cdot \right) \) represents the 2D local spatial condition. Intuitively, the context condition facilitates the modeling of global information, while the spatial condition enhances the capacity for local pixel-wise modeling. By appropriately combining spatial and context conditions, SCeLU demonstrates adaptability to complex visual layouts in various image recognition tasks. By simply changing the activation function, experiments conducted on ImageNet demonstrate a significant enhancement and robustness of SCeLU, particularly for small models, and some enhancements under partially highly optimized large models. Furthermore, our novel SCeLU seamlessly extends to object detection and semantic segmentation tasks, underscoring its effectiveness as an effective alternative in various visual recognition tasks. Our model is open-sourced at https://github.com/YunDuanFei/SCeLU .