Improving Segmentation Models for AR-guided Liver Surgery using Synthetic Images

AR-guided open liver surgery is a field of intense research. However, due to the lack ofRGB-D videos of the surgery scene, there are not any solutions for automatic real-time tracking and registration of the virtual models to the patient’s anatomy, yet. We provide the first proof of concept for generating synthetic liver surgery images using surgery phantoms with a 3D print of a real liver. Thus, the RGB-D camera of an AR device captures realistic depth patterns. The RGB images of the phantom are enriched by realistic liver textures using image synthesis methods. We use these data to augment training data for RGB-D segmentation. Furthermore, we compare three common image synthesis methods that are based on generative adversarial networks (GANs) in demo setting for this purpose. We evaluate our synthetic data by measuring the performance of an RGB-D segmentation model for porcine liver images. Results showthatwe can outperform models trained only on real data by 3% to 4% when using a GauGAN approach. Furthermore, we observe biases due to overuse of synthetic data for augmentation factors higher than 50 %. Results propose a novel phantom-based concept for data synthesis in AR-guided surgery and serve as guidance for future technical improvements.