Automated Marker-Less Patient-to-Preoperative Medical Image Registration Approach Using RGB-D Images and Facial Landmarks for Potential Use in Computed-Aided Surgical Navigation of the Paranasal Sinus

Paranasal sinus surgery is an established treatment option for chronic rhinosinusitis. Because this surgery is performed inside the nasal cavity, where critical anatomical structures, such as optic nerves and pituitary glands, exist nearby, surgeons usually rely on computer-aided surgical navigation (CSN) to provide a wide field of view in the surgical site and to allow for precise control of surgical instruments. In the CSNs, it is essential to register the surgical site of the actual patient with the corresponding view from the preoperative computed tomography (CT) images. The traditional registration approaches are performed manually by the user or automatically by attaching fiducial markers on both the patient’s surgical site and preoperative CT images for every surgery before use. In this work, we propose an automated approach to register patient-to-preoperative CT image without fiducial markers. The proposed approach detected and extracted facial anatomical landmarks in 2D RGB images through the use of deep learning models. These landmarks were located in 3D facial mesh reconstructed from depth images by using unprojection and ray-marching algorithms. The facial landmark pairs acquired from the patient site and the preoperative CT images are then registered with singular value decomposition and iterative closet point algorithms. We demonstrate the registration capability of our approach using Microsoft HoloLens 2, a mixed reality head-mounted display because it facilitates the acquisition of RGB-depth images and the prototype development of in-situ visualization to illustrate how the CT images are properly registered on the target surgical site. We compared our automated marker-less registration approach to the manual counterpart using a facial phantom with three participants. The results show that our approach produces relatively good registration accuracy, with a marginal target registration error of 4.4 mm when compared to the manual counterpart.