Do-DETR: enhancing DETR training convergence with integrated denoising and RoI mechanism
摘要
As the pioneering work in Transformer-based object detection, DETR has attracted widespread attention and sparked a research trend since its inception. DETR’s global attention mechanism is novel in its architecture, but it takes a very long time to optimize and reach good performance. To address this issue, we introduce DO-DETR in this paper. Specifically, except for the Hungarian loss, we build a denoising module where noisy GT bounding boxes are inputted into the decoder, which trains the model to reconstruct the original boxes. This process significantly simplifies the complexity of bipartite graph matching, resulting in accelerated convergence. In the decoder part, we designed a multi-layer recurrent processing structure based on RoI, which helps the attention of DETR gradually and more accurately focus on foreground objects. Visual features are taken as glimpse features from the larger bounding box regions of RoIs in each processing stage based on the detection outcomes of the preceding stage. These glimpse features are then modeled together with the attention outputs from the previous stage, thereby alleviating the difficulty of global attention modeling. Under the ResNet-50 backbone, DO-DETR achieved the same AP (43.6) on the MSCOCO dataset in just 16 epochs, which vanilla DETR requires 500 epochs to achieve. Meanwhile, Deformable DETR took 50 epochs to achieve a similar performance. Our DO-DETR thus improved the convergence efficiency of Deformable DETR by 68%.