Scholarlink

Purpose <p>Video question answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation.</p> Methods <p>We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video–Text Encoder to fuse video and question features, capturing temporal cues like motion and tool–tissue interactions, which a fine-tuned LLM then decodes into coherent answers. To evaluate its performance, we curate REAL-Colon-VQA, a colonoscopic video dataset including motion questions and diagnostic attributes, including out-of-template questions with rephrased or semantically altered formulations to evaluate model robustness.</p> Results <p>Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models and remains competitive with a fine-tuned video VLM baseline. In particular, SurgViVQA improves over PitVQA by +9% on REAL-Colon-VQA and +9% on EndoVis18-VQA in Keyword Accuracy, while achieving the strongest overall lexical and semantic generation performance. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing.</p> Conclusion <p>SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

SurgViVQA: temporally grounded video question answering for surgical scene understanding

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月22日

查看全文
  • Mauro Orazio Drago,
  • Luca Carlini,
  • Pelinsu Celebi Balyemez,
  • Dennis Pierantozzi,
  • Chiara Lena,
  • Cesare Hassan,
  • Danail Stoyanov,
  • Elena De Momi,
  • Sophia Bano,
  • Mobarak I. Hoque

摘要

Purpose

Video question answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation.

Methods

We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video–Text Encoder to fuse video and question features, capturing temporal cues like motion and tool–tissue interactions, which a fine-tuned LLM then decodes into coherent answers. To evaluate its performance, we curate REAL-Colon-VQA, a colonoscopic video dataset including motion questions and diagnostic attributes, including out-of-template questions with rephrased or semantically altered formulations to evaluate model robustness.

Results

Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models and remains competitive with a fine-tuned video VLM baseline. In particular, SurgViVQA improves over PitVQA by +9% on REAL-Colon-VQA and +9% on EndoVis18-VQA in Keyword Accuracy, while achieving the strongest overall lexical and semantic generation performance. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing.

Conclusion

SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts.