Remote sensing image semantic segmentation faces multiple challenges: complex background environments, large variations in target scales, and significant intra-class differences. Existing semantic segmentation methods primarily focus on spatial domain feature extraction and fusion. This limitation restricts model performance in complex scenes. To tackle the aforementioned problems, we propose a cross-domain fusion network (CDFNet). First, we introduce a Dual-Domain Feature Representation (DDFR) module that leverages spatial and frequency domain features to refine global and fine-grained spatial target features. Second, we develop a Cross-Domain Fusion Module (CDFM) that achieves semantic matching across domains through selective feature fusion via cross-attention mechanisms. Finally, we design a Cross-Scale Interaction Module (CSIM) that models and aggregates multi-scale encoder features with CDFM outputs through correlation operations, enabling better identification and understanding of diverse target features. We performed comprehensive ablation studies and comparative tests on three benchmarks: ISPRS Vaihingen, ISPRS Potsdam, and LoveDA. Our method outperformed others, attaining mIoU scores of 84.33%, 87.84%, and 53.62% on the respective datasets. The code is located at https://anonymous.4open.science/r/CDFNet-05B3 .