Purpose Tissue tracking is critical for downstream tasks in robot-assisted surgery. The Sparse Efficient Neural Depth and Deformation (SENDD) model has previously demonstrated accurate and real-time sparse point tracking, but struggled with occlusion handling. This work extends SENDD to enhance occlusion detection and tracking consistency while maintaining real-time performance. Methods We use the Segment Anything Model2 (SAM2) [<CitationRef CitationID="CR1">1</CitationRef>] to detect and mask occlusions by surgical tools, and we develop and integrate into SENDD an Adaptive Multi-Flow Sparse Tracker (A-MFST) with forward–backward consistency metrics, to enhance occlusion and uncertainty estimation. A-MFST is an unsupervised variant of the Multi-Flow dense Tracker (MFT) [<CitationRef CitationID="CR2">2</CitationRef>]. Results We evaluate our approach on the STIR dataset [<CitationRef CitationID="CR3">3</CitationRef>] and demonstrate a significant improvement in tracking accuracy under occlusion, reducing average tracking errors by 12% in Mean Endpoint Error (MEE) and showing a 6% improvement in <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="11548_2025_3414_Article_IEq1.gif" Format="GIF" Height="20" Rendition="HTML" Resolution="72" Type="Linedraw" Width="27" /> </InlineMediaObject> <EquationSource Format="TEX">$ \delta _{\text {avg}}^{x} $</EquationSource> <EquationSource Format="MATHML"><math> <msubsup> <mi>δ</mi> <mrow> <mtext>avg</mtext> </mrow> <mi>x</mi> </msubsup> </math></EquationSource> </InlineEquation>, the averaged accuracy over thresholds of [4, 8, 16, 32, 64] pixels [<CitationRef CitationID="CR4">4</CitationRef>]. The incorporation of forward–backward consistency further improves the selection of optimal tracking paths, reducing drift and enhancing robustness. Notably, these improvements were achieved without compromising the model’s real-time capabilities. Conclusions Using A-MFST and SAM2, we enhance SENDD’s ability to track tissue in real-time, under instrument and tissue occlusions. Our approach improves tracking accuracy and reliability by integrating SAM2 for robust occlusion handling and employing forward–backward consistency for optimal frame selection. Experimental results on the STIR dataset demonstrate that A-MFST reduces tracking errors while preserving real-time performance, making it well suited for surgical applications. Future work will focus on further refining adaptive mechanisms to enhance robustness and computational efficiency.

A-MFST: adaptive multi-flow sparse tracker for real-time tissue tracking under occlusion

期刊论文
发表日期：

2025年5月22日

查看全文

Yuxin Chen,
Zijian Wu,
Adam Schmidt,
Septimiu E. Salcudean

摘要

Purpose

Tissue tracking is critical for downstream tasks in robot-assisted surgery. The Sparse Efficient Neural Depth and Deformation (SENDD) model has previously demonstrated accurate and real-time sparse point tracking, but struggled with occlusion handling. This work extends SENDD to enhance occlusion detection and tracking consistency while maintaining real-time performance.

Methods

We use the Segment Anything Model2 (SAM2) [1] to detect and mask occlusions by surgical tools, and we develop and integrate into SENDD an Adaptive Multi-Flow Sparse Tracker (A-MFST) with forward–backward consistency metrics, to enhance occlusion and uncertainty estimation. A-MFST is an unsupervised variant of the Multi-Flow dense Tracker (MFT) [2].

Results

We evaluate our approach on the STIR dataset [3] and demonstrate a significant improvement in tracking accuracy under occlusion, reducing average tracking errors by 12% in Mean Endpoint Error (MEE) and showing a 6% improvement in $ \delta _{\text {avg}}^{x} $ $δ_{avg}^{x}$ , the averaged accuracy over thresholds of [4, 8, 16, 32, 64] pixels [4]. The incorporation of forward–backward consistency further improves the selection of optimal tracking paths, reducing drift and enhancing robustness. Notably, these improvements were achieved without compromising the model’s real-time capabilities.

Conclusions

Using A-MFST and SAM2, we enhance SENDD’s ability to track tissue in real-time, under instrument and tissue occlusions. Our approach improves tracking accuracy and reliability by integrating SAM2 for robust occlusion handling and employing forward–backward consistency for optimal frame selection. Experimental results on the STIR dataset demonstrate that A-MFST reduces tracking errors while preserving real-time performance, making it well suited for surgical applications. Future work will focus on further refining adaptive mechanisms to enhance robustness and computational efficiency.