Whole slide image (WSI) segmentation plays a critical role in the precision medicine of colorectal cancer, as it enables detailed analysis of tumor morphology and microenvironment, which are essential for accurate diagnosis and treatment planning. However, this task faces two major challenges: (1) the gigapixel resolution of WSIs necessitates patch-based processing, leading to the loss of global contextual information; and (2) existing multi-resolution fusion methods suffer from the loss of positional and semantic information due to late-stage feature fusion, insufficient utilization of low-resolution contextual information, and high computational complexity, resulting in suboptimal segmentation performance. To address these issues, we propose a novel multi-stage multi-resolution fusion framework. First, we introduce a dual-branch encoder, where the frozen low-resolution branch efficiently captures global contextual information, while the trainable high-resolution branch preserves fine-grained spatial details. Second, we specifically design a decoder that first optimizes low-resolution features to mitigate the imbalance between dual-branch features, followed by stage-wise feature fusion to fully leverage the wide field of view of the low-resolution branch for enhanced segmentation results. Additionally, we incorporate a multi-scale feature fusion and optimization module to deeply refine features and improve the model’s segmentation performance. Our method demonstrates significant advantages in multi-class semantic segmentation tasks on a colorectal cancer WSI dataset. The code and model weights will be made publicly available to support clinical decision-making.