Unleash the Power of 2D Pre-trained Model for 3D T1-weighted Brain MRI Inpainting

The performance of current brain image analysis pipelines can dramatically degrade when handling images with pathologies (e.g., tumors). To address this issue, one solution is using image inpainting to generate synthetic healthy tissues within regions of abnormality. In recent years, image inpainting algorithms have achieved significant improvements due to the rapid development of Convolutional Neural Networks (CNNs). However, brain image inpainting task remains an open challenge because of the insufficient training data and the diversity of brain abnormalities. In this paper, we try to leverage a 2D inpainting model pre-trained on a large dataset of natural images for the 3D brain inpainting task. We effectively fine-tune the 2D inpainting model using brain MRI slices from different views and fuse slice-level predictions to form a volume-level inpainted image. A post-processing strategy is proposed based on the morphological closing operation to refine fused results. Experiments show that the 2D pre-trained inpainting model has a strong generalization ability even without fine-tuning and performance can be further improved with fine-tuning and post-processing applied to the full volume.