TVFNet: text and visual attention feature fusion network for multi-lesion segmentation of diabetic retinopathy
摘要
Diabetic retinopathy (DR) is asignificant cause of blindness in the global population. Automatic and accurate lesion segmentation helps to assist ophthalmologists in DR diagnosis and grading. However, multiple lesion segmentation is full of challenges due to the inter-class similarities, intra-class differences and complex structures. Although previous research has made some progress, most works ignored the multi-source information of textual labels and visual features. This paper proposes a text and visual attention feature fusion network for multi-lesion segmentation of diabetic retinopathy. The proposed model incorporates textual labels of the number and scale of lesions with visual features by element-wise multiplication for multi-source information fusion. First, ResNeXt-50 is used to extract features from retinal images and classify lesion types and scales. Then, the feature enhancement module (FEM) adopts the triple attention mechanism composed of channel attention, spatial attention and pixel attention to focus on important lesion regions. Next, the text label attention module (TLAM) utilizes byte-pair encoding and element-wise multiplication to adaptively fuse textual label information with visual features as supervision for the decoder. Finally, the multi-scale feature fusion module (MSFM) adopts 1×1 convolution and sigmoid operation to generate lesion segmentation results. Extensive experimental results on four public datasets of IDRiD, DDR, FGADR and E-ophtha well demonstrate the superiority of our model, which achieves competitive performance compared to state-of-the-arts. It has better accuracy and robustness, which can overcome adverse interference from other fundus tissues and background noise for precise multiclass lesion segmentation.