Segmentation-guided cross-modal spine image generation with disjoint patient sets

Data-driven medical imaging tasks rely on a large amount of data, and specific modal data may be insufficient in training. Cross-modal image synthesis is one of the solutions. However, most of the existing generation methods still require both modal data for the same patient. If disjoint patient data sets can be used, cross-modal generation methods can be made more flexible and practical. Therefore, we aim to generate cross-modal spinal images between MR and CT with disjoint patient sets. We propose a CycleGAN-based network to intersynthesize disjoint MR and CT spine datasets. To address the problem that the region of interest is prone to distortion during generation, the network uses segmentation guidance mechanism to enhance the relative position and posture of the spine. In addition, a self-attention mechanism is introduced in the generator to take into account the overall context information. Our method was evaluated and compared with other methods in both MR and CT cross-modal generation tasks. Experiments are mainly conducted on imaging quality, segmentation performance improvement, potential feature distribution and visualization of guided attention. Ablation studies were performed to evaluate the module gain, the generality of the guidance model, and the value of the guidance weight. Results show that our method achieved better quantitative and qualitative results than the baseline. In medical image tasks such as segmentation, synthetic images by our method can assist the task to achieve better performance.