<p>Diabetic retinopathy (DR) is asignificant cause of blindness in the global population. Automatic and accurate lesion segmentation helps to assist ophthalmologists in DR diagnosis and grading. However, multiple lesion segmentation is full of challenges due to the inter-class similarities, intra-class differences and complex structures. Although previous research has made some progress, most works ignored the multi-source information of textual labels and visual features. This paper proposes a text and visual attention feature fusion network for multi-lesion segmentation of diabetic retinopathy. The proposed model incorporates textual labels of the number and scale of lesions with visual features by element-wise multiplication for multi-source information fusion. First, ResNeXt-50 is used to extract features from retinal images and classify lesion types and scales. Then, the feature enhancement module (FEM) adopts the triple attention mechanism composed of channel attention, spatial attention and pixel attention to focus on important lesion regions. Next, the text label attention module (TLAM) utilizes byte-pair encoding and element-wise multiplication to adaptively fuse textual label information with visual features as supervision for the decoder. Finally, the multi-scale feature fusion module (MSFM) adopts 1×1 convolution and sigmoid operation to generate lesion segmentation results. Extensive experimental results on four public datasets of IDRiD, DDR, FGADR and E-ophtha well demonstrate the superiority of our model, which achieves competitive performance compared to state-of-the-arts. It has better accuracy and robustness, which can overcome adverse interference from other fundus tissues and background noise for precise multiclass lesion segmentation.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TVFNet: text and visual attention feature fusion network for multi-lesion segmentation of diabetic retinopathy

  • Yanfei Guo,
  • Yuanke Zhang,
  • Fei Ma,
  • Jing Meng,
  • Shasha Yuan,
  • Jindong Sun

摘要

Diabetic retinopathy (DR) is asignificant cause of blindness in the global population. Automatic and accurate lesion segmentation helps to assist ophthalmologists in DR diagnosis and grading. However, multiple lesion segmentation is full of challenges due to the inter-class similarities, intra-class differences and complex structures. Although previous research has made some progress, most works ignored the multi-source information of textual labels and visual features. This paper proposes a text and visual attention feature fusion network for multi-lesion segmentation of diabetic retinopathy. The proposed model incorporates textual labels of the number and scale of lesions with visual features by element-wise multiplication for multi-source information fusion. First, ResNeXt-50 is used to extract features from retinal images and classify lesion types and scales. Then, the feature enhancement module (FEM) adopts the triple attention mechanism composed of channel attention, spatial attention and pixel attention to focus on important lesion regions. Next, the text label attention module (TLAM) utilizes byte-pair encoding and element-wise multiplication to adaptively fuse textual label information with visual features as supervision for the decoder. Finally, the multi-scale feature fusion module (MSFM) adopts 1×1 convolution and sigmoid operation to generate lesion segmentation results. Extensive experimental results on four public datasets of IDRiD, DDR, FGADR and E-ophtha well demonstrate the superiority of our model, which achieves competitive performance compared to state-of-the-arts. It has better accuracy and robustness, which can overcome adverse interference from other fundus tissues and background noise for precise multiclass lesion segmentation.