Hybrid 3D Medical Image Segmentation Using CNN and Frequency Transformer Fusion

Medical image segmentation poses a significant challenge, particularly with 3D images. This study presents a novel hybrid network for 3D medical image segmentation, which sequentially integrates convolutional neural networks (CNNs) and transformer models. Structured with an encoder-decoder architecture, the network comprises a frequency transformer branch and a variational autoencoder branch. Initially, the CNN encoder extracts the features of volumetric spatial from the input 3D image. These features are subsequently converted into vectors and incorporated into a frequency transformer. This transformer employs fast Fourier transforms instead of self-attention to mitigate computational complexity. Significantly, this research marks the inaugural utilization of the Fourier transform on transformers for image segmentation employing 3D datasets. The network’s efficacy was evaluated on kidney, tumor, and brain 3D datasets. It achieved remarkable results, such as a 97.54% dice score and a 95%-Hausdorff distance (95%-HD) of 3.26 mm for kidney segmentation, and an 86.92% dice score with a 95%-HD of 2.57 mm for brain tumor segmentation. These results underscore the network’s potential for accurate and reliable segmentation in diverse medical imaging applications.