<p>Retinal image captioning is an crucial task in medical image analysis. However, many existing models struggle to generate accurate predictions due to a lack of data or ineffective cross-modal alignment techniques. These limitations hinder the model’s ability to achieve high performance, which is required in the medical domain. A novel approach is proposed by integrating a Differential Transformer in order to enhance the model performance. In addition, a weight-sharing strategy is applied to keyword embedding layer, token embedding layer, and language head to reduce the model size and improve the training process. Experimental results show that the proposed model demonstrates strong flexibility and adaptability when integrated with various Convolutional Neural Network (CNN) architectures as visual feature extractors, showcasing its robustness across different backbone networks. Extensive experiments on benchmark retinal image datasets validate the effectiveness of our approach, with significant improvements observed in standard evaluation metrics compared to recent state-of-the-art methods. These results highlight the potential of our model to serve as a foundation for future advancements in medical image captioning, paving the way for more accurate and efficient automated diagnostic systems.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Differential attention for retinal image captioning via cross-modal integration

  • Nguyen Ha Hieu,
  • Le Van Thanh,
  • Luong Thi Hong Lan,
  • Hai Van Pham

摘要

Retinal image captioning is an crucial task in medical image analysis. However, many existing models struggle to generate accurate predictions due to a lack of data or ineffective cross-modal alignment techniques. These limitations hinder the model’s ability to achieve high performance, which is required in the medical domain. A novel approach is proposed by integrating a Differential Transformer in order to enhance the model performance. In addition, a weight-sharing strategy is applied to keyword embedding layer, token embedding layer, and language head to reduce the model size and improve the training process. Experimental results show that the proposed model demonstrates strong flexibility and adaptability when integrated with various Convolutional Neural Network (CNN) architectures as visual feature extractors, showcasing its robustness across different backbone networks. Extensive experiments on benchmark retinal image datasets validate the effectiveness of our approach, with significant improvements observed in standard evaluation metrics compared to recent state-of-the-art methods. These results highlight the potential of our model to serve as a foundation for future advancements in medical image captioning, paving the way for more accurate and efficient automated diagnostic systems.