Lightweight and efficient model design is a significant trend in the field of deep learning. For scene text recognition (STR) task, most transformer-based methods have achieved remarkable accuracy. However, transformers have been criticized for their quadratic growth in computational overhead with sequence length. Mamba, as a well-designed solution for sequence modeling, has not been explored for STR. In this paper, we design a Swin-Mamba which is a highly parallelized solution based on Mamba for local window interaction and cross window connection. Based on it, we propose a novel mamba-based model for STR, called MambaSTR, striving to achieve comparable accuracy while maintaining fewer parameters and lower inference overhead. MambaSTR consists of a Swin-Mamba-based dual-branch encoder and a simple mamba decoder. Each encoder branch performs a series of local and global interaction on image tokens according to different scanning directions. Mixture-of-Experts strategy is adopted to dynamically fuse the information from different directions. Extensive experiments demonstrate that MambaSTR achieves SOTA accuracy on multiple datasets. The code is available at https://github.com/bozhou507/MambaSTR .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MambaSTR: Scene Text Recognition with Masked State Space Model

  • Bo Zhou,
  • Anna Zhu,
  • JunJie Yang

摘要

Lightweight and efficient model design is a significant trend in the field of deep learning. For scene text recognition (STR) task, most transformer-based methods have achieved remarkable accuracy. However, transformers have been criticized for their quadratic growth in computational overhead with sequence length. Mamba, as a well-designed solution for sequence modeling, has not been explored for STR. In this paper, we design a Swin-Mamba which is a highly parallelized solution based on Mamba for local window interaction and cross window connection. Based on it, we propose a novel mamba-based model for STR, called MambaSTR, striving to achieve comparable accuracy while maintaining fewer parameters and lower inference overhead. MambaSTR consists of a Swin-Mamba-based dual-branch encoder and a simple mamba decoder. Each encoder branch performs a series of local and global interaction on image tokens according to different scanning directions. Mixture-of-Experts strategy is adopted to dynamically fuse the information from different directions. Extensive experiments demonstrate that MambaSTR achieves SOTA accuracy on multiple datasets. The code is available at https://github.com/bozhou507/MambaSTR .