Streamlining stereo-differentiable rendering for marker-free real-time tracking of surgical robots

We evaluate stereo-differentiable rendering-based pose estimation for marker-free real-time surgical robots tracking, mitigating occlusion-prone marker-based tracking in cluttered surgical environments, potentially improving safety, reducing setup times, and enabling intelligent multi-robot interaction.

This work extends the differentiable rendering-based markerless robot pose estimation framework roboreg for online real-time dynamic tracking in two ways. (i) Sequential optimisation propagates pose estimates across consecutive frames, with motion-adaptive hyperparameter tuning balancing convergence and precision during estimation. (ii) Integrate CUDA stream parallelisation for segmentation and the optimisation steps and combines it with CUDA-graph accelerated segmentation. We collect 38 displacement video sequence datasets with unobstructed robot and 5 occluded-robot dataset with static start/end ground-truth pose calibrations and dynamic marker-based reference tracking in between for accuracy evaluation under different scenarios.

Real-time localisation at 30 fps for 1080p video sequence is achieved, accelerating from 14 fps in the vanilla roboreg, thereby matching the camera frame rate. Near-1 cm accuracy is demonstrated, with 1.7 cm translational and 0. \(6^\circ \) $6^{\circ }$ rotational error against static ground-truth pose calibration; and with 1.2 cm average 3D error across 27,460 frames against a marker-based reference standard (1.53 cm in over 1242 frames in occlusion evaluation). Our method outperforms FoundationPose by 11% (63% in occlusion dataset) in dynamic estimation and 250% in static estimation, while achieving \(6 \times \) $6\times$ faster inference.

We demonstrate real-time high-resolution marker-free tracking of surgical robots through stereo-differentiable rendering. Localisation accuracy performed on par with marker-based approaches and improved upon foundational baselines.