Scene text detection is a key technology in computer vision and is crucial for applications such as autonomous driving, intelligent navigation, and image retrieval. In recent years, methods based on large-scale contrastive language-image pre-training (CLIP) have made significant progress in this field. However, while existing CLIP-based methods have improved performance, they also introduce significant computational overhead. To address this issue, we propose MixFormer (A novel transformer approach incorporating textual features), which uses the CLIP text encoder as an independent auxiliary branch and designs a multi-level text-image fusion mechanism. This approach efficiently integrates CLIP’s semantic capabilities while only increasing parameters by 5.2%, significantly enhancing text feature representation. Experiments demonstrate that MixFormer achieves a F1 score of 90.0% on the CTW1500 dataset, setting a new state-of-the-art record, and validates that this method can significantly improve detection accuracy without significantly increasing computational costs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MixFormer: A Cross-Modal Transformer for Arbitrary-Shaped Scene Text Detection

  • Yaolin Weng,
  • Chuanlong Liu,
  • Miaomiao Xu,
  • Mieradilijiang Maimaiti,
  • Wushour Silamu

摘要

Scene text detection is a key technology in computer vision and is crucial for applications such as autonomous driving, intelligent navigation, and image retrieval. In recent years, methods based on large-scale contrastive language-image pre-training (CLIP) have made significant progress in this field. However, while existing CLIP-based methods have improved performance, they also introduce significant computational overhead. To address this issue, we propose MixFormer (A novel transformer approach incorporating textual features), which uses the CLIP text encoder as an independent auxiliary branch and designs a multi-level text-image fusion mechanism. This approach efficiently integrates CLIP’s semantic capabilities while only increasing parameters by 5.2%, significantly enhancing text feature representation. Experiments demonstrate that MixFormer achieves a F1 score of 90.0% on the CTW1500 dataset, setting a new state-of-the-art record, and validates that this method can significantly improve detection accuracy without significantly increasing computational costs.