Purpose The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery. Methods We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a Yolo-NAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence. Results Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$284\,\upmu \hbox {m}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>284</mn> <mspace width="0.166667em" /> <mi mathvariant="normal">μ</mi> <mtext>m</mtext> </mrow> </math></EquationSource> </InlineEquation> (OPMI only) to <InlineEquation ID="IEq2"> <EquationSource Format="TEX">$33\,\upmu \hbox {m}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>33</mn> <mspace width="0.166667em" /> <mi mathvariant="normal">μ</mi> <mtext>m</mtext> </mrow> </math></EquationSource> </InlineEquation> (multimodal). Conclusion Feature fusion of multimodal imaging can enhance multitask prediction accuracy compared to single-modality processing, and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multimodal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.

Toward comprehensive real-time scene understanding in ophthalmic surgery through multimodal image fusion

期刊论文
出版模式：

开放获取
发表日期：

2026年5月21日

查看全文

Nikolo Rohrmoser,
Ghazal Ghazaei,
Michael Sommersperger,
Nassir Navab

摘要

Purpose

The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery.

Methods

We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a Yolo-NAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence.

Results

Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from $284\,\upmu \hbox {m}$ $284 μ m$ (OPMI only) to $33\,\upmu \hbox {m}$ $33 μ m$ (multimodal).

Conclusion

Feature fusion of multimodal imaging can enhance multitask prediction accuracy compared to single-modality processing, and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multimodal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.