Abstract: Generation of Synthetic 3D Data using Simulated MR Examinations in Augmented Reality

In the future, medical imaging devices such as computed tomography (CT) and magnetic resonance (MR) are expected to become increasingly autonomous. Therefore, design criteria and workflow of such devices will change substantially. Moreover, sensor data of the system are required to develop scene understanding algorithms to support and guide the user. Data availability might be a critical factor due to patient privacy issues, high cost of labelling data or impossibility to acquire it in dangerous situations. In this work, we present an approach to generate synthetic 3D point cloud data from a simulated MR examination experienced on the Microsoft Hololens2. The complete workflow of an MR examination using a virtual, autonomousMRscanner is reproduced in an AR scene. The user can interact with an avatar of the patient via voice commands, select a procedure in a GUI or position a coil. The user is recorded by a system of active stereo vision RGBD-cameras while interacting with AR elements. A registration routine of the AR scene and the RGBD-cameras is described, and accuracy measurements are provided. The real point clouds are fused with virtually generated point clouds from the AR scene. These point clouds are completely labelled and 3D bounding boxes of the objects as well as the rotation and translation of their corresponding CAD models are saved. Our approach can be used to generate synthetic depth data such as a real depth camera would see once the system – or even a system that does not yet physically exist - is built and installed on-site [1].