Tumor segmentation with multi-modal images is an important work in medical image analysis. The existing multi-modal medical image segmentation models achieve the alignment of the feature between different modal data by statistical methods or attention mechanisms. However, these methods may lead to the loss of modal-specific features of original modality. Therefore, this paper proposes a multi-modal medical image segmentation algorithm, DFSegmentation, which aims to decouple the modal-shared features and modal-specific features of different modal through their correlation in frequency domain. At the same time, inversible neural network and frequency-domain attention mechanism are used to solve the problem of high-frequency information loss and frequency-domain information fusion. We validate the effectiveness of our model through different tasks and compare it with other multimodal medical image segmentation models. The results show that our model can achieve better results with specific boundary or small scale. The code is available at https://github.com/LordofPupil/DFSegmentation .