LMTextSpotter: Towards Better Scene Text Spotting with Language Modeling in Transformer

End-to-end text spotting aims to build a unified framework for scene text detection and recognition. Recently, the DETR-like framework has attracted great attention. However, most of such studies merely focused on promoting the collaboration between two sub-tasks, without considering the importance of linguistic information in scene text. In this paper, we propose a novel end-to-end text spotting model, termed as LMTextSpotter, which introduces language modeling capability into the DETR-like text spotting framework. Specifically, we add an extra LM recognition branch besides the detection and commonly used CTC recognition branch in most existing frameworks, which is trained with the cross-entropy loss in autoregression manner. We backtrack the viterbi path given by CTC recognition branch to get an alignment between the queries and characters, so as to train LM recognition branch without relying on the character level annotations. After training phase, the LM recognition branch can predict the character classes independently of the CTC recognition branch. Besides, we exploit the position priors provided by rotated bounding boxes and adopt a task-specific query strategy for further improving the performance. Comparison results demonstrate the superiority of our LMTextSpotter over previous state-of-the-art methods. Ablation studies also prove the effectiveness of each component.