Ultra-wide-field (UWF) fundus photography provides a comprehensive view of the retina, enabling diabetic retinopathy (DR) detection in peripheral regions often missed in traditional imaging techniques. However, the lack of large-scale public UWF datasets hinders related deep learning-based DR detection research. Moreover, models trained on conventional classification targets frequently lack interpretability in their predictions or produce results that are misaligned with clinical criteria. To address these challenges, we first compile a dataset (UWF-DR) comprising over 5,000 UWF images, each graded for DR and annotated with lesion presence. Utilizing the annotations with clinical guidelines and GPT-4o’s vision-language capabilities, we generate reasoning-enhanced image captions that mirror the decision-making processes of ophthalmologists. An instruction dataset is constructed based on these captions and augmented with sub-task instructions. We fine-tune a multi-modal large language model on UWF-DR to generate reasoning for DR detection. Experimental results demonstrate the benefit of reasoning-based enhancement, showcasing superior grading performance and high alignment with clinical criteria. The dataset is available at UWF-DR .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Reasoning-Enhanced Vision-Language Model for Interpretable Diabetic Retinopathy Detection in Ultra-Wide-Field Fundus Images

  • Zhenyu Tang,
  • Lingzhi Chen,
  • Lilong Wang,
  • Yankai Jiang,
  • Jun Li,
  • Xiaosong Wang

摘要

Ultra-wide-field (UWF) fundus photography provides a comprehensive view of the retina, enabling diabetic retinopathy (DR) detection in peripheral regions often missed in traditional imaging techniques. However, the lack of large-scale public UWF datasets hinders related deep learning-based DR detection research. Moreover, models trained on conventional classification targets frequently lack interpretability in their predictions or produce results that are misaligned with clinical criteria. To address these challenges, we first compile a dataset (UWF-DR) comprising over 5,000 UWF images, each graded for DR and annotated with lesion presence. Utilizing the annotations with clinical guidelines and GPT-4o’s vision-language capabilities, we generate reasoning-enhanced image captions that mirror the decision-making processes of ophthalmologists. An instruction dataset is constructed based on these captions and augmented with sub-task instructions. We fine-tune a multi-modal large language model on UWF-DR to generate reasoning for DR detection. Experimental results demonstrate the benefit of reasoning-based enhancement, showcasing superior grading performance and high alignment with clinical criteria. The dataset is available at UWF-DR .