Scholarlink

Purpose: <p>Future human-in-the-loop surgical visual question answering (VQA) requires detecting unreliable answers before use. In this preclinical methodological study, we distinguish answer consistency from question validity: A model may generate mutually consistent answers that still fail to address the clinical question.</p> Methods: <p>We propose question-aligned semantic nearest neighbor entropy (QA-SNNE), a black-box failure-detection score that adds question-answer alignment to semantic entropy through bilateral gating. QA-SNNE modulates pairwise similarities among sampled answers by their question relevance, using embedding, entailment, or cross-encoder alignment. To assess robustness under language variation, we construct an out-of-template rephrased version of a benchmark surgical VQA dataset, in which only the question wording is modified while images and ground-truth answers remain unchanged.</p> Results: <p>We evaluate QA-SNNE on five VQA models across two benchmark surgical VQA datasets, considering both zero-shot and parameter-efficient fine-tuned settings, including out-of-template questions. QA-SNNE improves AUROC on EndoVis18-VQA for two of three zero-shot models in-template (e.g., +15% for Llama3.2 and +21% for Qwen2.5) and achieves up to +8% AUROC relative improvement among zero-shot models under out-of-template rephrasing, with mixed results on external validation.</p> Conclusion: <p>By making uncertainty estimation explicitly question-aware, QA-SNNE provides a practical and model-agnostic safeguard for preclinical surgical VQA evaluation. Linking semantic uncertainty to question relevance enables more reliable failure detection under variation in question phrasing.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

When to trust the answer: question-aligned semantic nearest neighbor entropy for safer surgical VQA

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月17日

查看全文
  • Luca Carlini,
  • Dennis Pierantozzi,
  • Mauro Orazio Drago,
  • Chiara Lena,
  • Cesare Hassan,
  • Elena De Momi,
  • Danail Stoyanov,
  • Sophia Bano,
  • Mobarak I. Hoque

摘要

Purpose:

Future human-in-the-loop surgical visual question answering (VQA) requires detecting unreliable answers before use. In this preclinical methodological study, we distinguish answer consistency from question validity: A model may generate mutually consistent answers that still fail to address the clinical question.

Methods:

We propose question-aligned semantic nearest neighbor entropy (QA-SNNE), a black-box failure-detection score that adds question-answer alignment to semantic entropy through bilateral gating. QA-SNNE modulates pairwise similarities among sampled answers by their question relevance, using embedding, entailment, or cross-encoder alignment. To assess robustness under language variation, we construct an out-of-template rephrased version of a benchmark surgical VQA dataset, in which only the question wording is modified while images and ground-truth answers remain unchanged.

Results:

We evaluate QA-SNNE on five VQA models across two benchmark surgical VQA datasets, considering both zero-shot and parameter-efficient fine-tuned settings, including out-of-template questions. QA-SNNE improves AUROC on EndoVis18-VQA for two of three zero-shot models in-template (e.g., +15% for Llama3.2 and +21% for Qwen2.5) and achieves up to +8% AUROC relative improvement among zero-shot models under out-of-template rephrasing, with mixed results on external validation.

Conclusion:

By making uncertainty estimation explicitly question-aware, QA-SNNE provides a practical and model-agnostic safeguard for preclinical surgical VQA evaluation. Linking semantic uncertainty to question relevance enables more reliable failure detection under variation in question phrasing.