The advancement in generative artificial intelligence (GenAI) in medicine, involving the use of multimodal AI models (narratives to text and image to text) and workflow tools play a crucial role in assisting clinicians towards understanding health trends, answering clinical questions, personalizing treatment, and compiling medical case reports. These advancements are only possible due to the development in several technical areas like meta-transformers, large-scale medical datasets, image captioning, optical character recognition and new loss functions. The rapid demand for these GenAI is due to clinician burnout, growing interlinked medical knowledge, staff shortages and an aging population. However, the current content management for clinical case report generation often lacks built-in tools for generating detailed image descriptions and relevant hash tags, leading to time-consuming manual work and potential inconsistencies. This short paper investigates the effectiveness of learning from two fine-tuned clinical vision-language models (BLIP-2 and Florence-2) for grounding textual descriptions to image regions that can contribute to compiling a clinical case report. Our investigation to learn from these two models is based on a popular Roco-2 Image and Text medical dataset. Our comparison shows the superiority of Florence-2 over BLIP-2

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Harnessing Image to Text Models for Clinical Case Report Generation

  • Sabah Mohammed,
  • Jinan Fiaidhi

摘要

The advancement in generative artificial intelligence (GenAI) in medicine, involving the use of multimodal AI models (narratives to text and image to text) and workflow tools play a crucial role in assisting clinicians towards understanding health trends, answering clinical questions, personalizing treatment, and compiling medical case reports. These advancements are only possible due to the development in several technical areas like meta-transformers, large-scale medical datasets, image captioning, optical character recognition and new loss functions. The rapid demand for these GenAI is due to clinician burnout, growing interlinked medical knowledge, staff shortages and an aging population. However, the current content management for clinical case report generation often lacks built-in tools for generating detailed image descriptions and relevant hash tags, leading to time-consuming manual work and potential inconsistencies. This short paper investigates the effectiveness of learning from two fine-tuned clinical vision-language models (BLIP-2 and Florence-2) for grounding textual descriptions to image regions that can contribute to compiling a clinical case report. Our investigation to learn from these two models is based on a popular Roco-2 Image and Text medical dataset. Our comparison shows the superiority of Florence-2 over BLIP-2