Medical Referring Image Segmentation: Addressing Multi-Lesion Reference and Annotation Uncertainty via Vision-Language Fusion
摘要
We introduce a novel method MedRIS for the challenging task of Medical Referring Image Segmentation (Medical RIS): segment the lesions described in a medical report. MedRIS addresses two major challenges: (I) a single text description may correspond to multiple lesions (one-to-many reference); (II) annotation may be uncertain due to complex appearance of lesions and subjectivity of annotators. To solve (I), mask self-augmentation separates the original mask into independent or combinations of lesions. To solve (II), ChatGPT and organ position matching are leveraged to reformulate textual descriptions into content-diverse sentences that are consistent with the augmented masks. Organ-related positional attention combined with real-world medical knowledge focuses the model on critical image areas. Exponential moving average (EMA) and local annotation correction are used to suppress the effect of noisy annotations during training. Extensive experiments on public and in house chest X-ray datasets show that MedRIS outperforms the state-of-the-art methods by a large margin.