A Unified CNN-ViT Network with a Feature Distribution Strategy for Multi-modal Missing MRI Sequences Imputation

Multi-modal magnetic resonance imaging (MRI) is of great clinical use for disease assessment and diagnosis, as it provides comprehensive and complementary information through multiple contrasts. However, due to potential obstacles of the scanning process and the patient’s physical condition, the available scans of each subject may vary. Here, we propose a unified adversarial network based on the convolution neural network (CNN) and vision transformer (ViT) for missing MRI image synthesis in any input-output image configurations. The purpose of our network design is to develop a robust and efficient way to integrate the local information capturing ability of the convolutional operation and global contextual sensitivity of the multi-head self-attention (MSA) mechanism. Specifically, we employ a u-shape network as our generator and mix the convolutional path and the MSA path in a parallel manner at each network stage, the mixer is in place of the original MSA block of the canonical transformer, this design enables each layer to learn both global and local information simultaneously and takes advantage of the general ViT architecture. Furthermore, since these two branches have different frequency information preference, possessing all channels in each branch inevitably renders feature redundancy and introduces extra artifacts, accordingly, we adopt a channel splitting strategy that splits input features channel-wise and feed to each branch separately, meanwhile, considering each network stage has different desire of the high- and low-frequency information, we also explore various channel distribution ratios at each network stage. Our proposed method demonstrates reliable synthesis for healthy tissues and heterogeneous enhancement on BraTS2021 dataset. Furthermore, compare with four state-of-the-art methods including convolution-based, MSA-based and hybrid models in the MRI sequence synthesis field, our approach outperforms these methods quantitatively and qualitatively. Therefore, our method has the potential to be used as a candidate for medical image synthesis.