Trans-Window Panoramic Impasto for Online Tissue Deformation Recovery

Deformation recovery from laparoscopic images benefits many downstream applications like robot planning, intraoperative navigation and surgical safety assessment. We define tissue deformation as time-variant surface structure and displacement. Besides, we also pay attention to the surface strain, which bridges the visual observation and the tissue biomechanical status, for which continuous pointwise surface mapping and tracking are necessary. Previous SLAM-based methods cannot cope with instrument-induced occlusion and severe scene deformation, while the neural field-based ones are offline and scene-specific, which hinders their application in continuous mapping. Moreover, neither approach meets the requirement of continuous pointwise tracking. To overcome these limitations, we assume a deformable environment and a movable window through which an observer depicts the environment’s 3D structure on a canonical canvas as maps in a process named impasto. The observer performs panoramic impasto for the currently and previously observed 3D structure in a two-step online approach: optimization and fusion. The optimization of the maps compensates for the error in the observation of the structure and the tracking by preserving spatiotemporal smoothness, while the fusion is for merging the estimated and the newly observed maps by ensuring visibility. Experiments were conducted using ex vivo and in vivo stereo laparoscopic datasets where tool-tissue interaction occurs and large camera motion exists. Results demonstrate that the proposed online method is robust to instrument-induced occlusion, capable of estimating surface strain, and can continuously reconstruct and track surface points regardless of camera motion. Code is available at: https://github.com/bmpelab/trans_window_panoramic_impasto.git .