Discriminatively Matched Part Tokens for Pointly Supervised Instance Segmentation
摘要
The self-attention mechanism of vision transformer has demonstrated remarkable superiority in fully-supervised instance segmentation. However, when supervision is constrained to a single point per instance, self-attention struggles to capture semantic variations across object parts, particularly for objects with substantial deformations and diverse appearances. In this study, we propose discriminatively matched part tokens (DMPT), to endow self-attention the capability of handling significant semantic variation for pointly supervised instance segmentation. DMPT first allocates a token for each object part by finding a semantic extreme point, and then introduces part classifiers with deformable constraint to re-estimate part tokens which are utilized to guide and enhance the fine-grained localization capability of the self-attention mechanism. Through iterative optimization, DMPT matches the most discriminative tokens which facilitate capturing fine-grained part semantics and activating full object extent. Extensive experiments on PASCAL VOC and MS COCO segmentation datasets show that DMPT respectively outperforms the state-of-the-art pointly supervised method by 2.0% mAP