Spatial Context Awareness in Surgery Through Sound Source Localization

Context awareness and scene understanding is an integral component for the development of intelligent systems in computer-aided and robotic surgery. While most systems primarily utilize visual data for scene understanding, recent proof-of-concepts have showcased the potential of acoustic signals for the detection and analysis of surgical activity that is associated with typical noise emissions. However, acoustic approaches have not yet been effectively employed for localization tasks in surgery, which are crucial to obtain a comprehensive understanding of a scene. In this work, we introduce the novel concept of Sound Source Localization (SSL) for surgery which can reveal acoustic activity and its location in the surgical field, therefore providing insight into the interactions of surgical staff with the patient and medical equipment. We show the potential of this concept by interpreting sound activity heatmaps using an acoustic camera in two proof-of-concept localization tasks, an object detection task for surgical sawing and a keypoint detection task for surgical chiseling. We achieve an AP at 0.5 IoU of 86.07% for the object detection task and a mean euclidean distance of \(13.70\pm 14.65\) px at an image resolution of 1100 \(\,\times \,\) 825 px for the keypoint detection task. Based on these results, we believe that the localization of acoustic events has great potential for surgical scene understanding, opening up many new research directions for multimodal sensing solutions in the operating room of the future. To the best knowledge of the authors this is the first work that proposes to leverage SSL in the medical context.