With the development of imaging technology, multimodal data has been applied to the processing and analysis of medical images. However, domain differences between different modalities introduce a significant amount of interference information, resulting in significant variations in pixel intensities at the same location, thereby affecting the segmentation accuracy. In this paper, we propose a lightweight multimodal segmentation network based on the Siamese network architecture, called SiamSegNet. We introduce a cross-modal generation strategy to explore the potential consistency of visual representations between different modalities. The aggregation of the cross-modal generation strategy and the Siamese network aims to reduce the domain differences between different modalities and capture the complementary semantic information that contributes to the segmentation of key regions. To further investigate the fusion performance of cross-modal fusion methods for modalities with significant visual representation differences, we evaluate our method on multiple medical datasets. On the same benchmarks, our method outperforms state-of-the-art segmentation frameworks.