Colonoscopy is the most effective screening method for detecting colorectal cancer (CRC) and its precursors lesions, such as polyps. However, colonoscopy images have complex backgrounds and are often affected by lighting conditions, resulting in substantial background noise. Additionally, polyps vary in sizes and shapes, with some having unclear boundaries, leading to the difficulty in accurate localization. Most of the existing polyp segmentation methods rely on convolutional neural networks (CNNs) or Transformers to extract features, while CNNs-based models are focusing on modeling local information, and Transformer-based models are focusing on modeling global information. Despite these efforts, current works fail to fully leverage the strengths of both types of approaches, and can not achieve a complementary integration of local and global information. These limitations hinder models from effectively utilizing long-range contextual information in the images, and lead to inaccurate boundary localization and blurry segmentation predictions. To address these limitations, we propose a novel polyp segmentation method named CM-Net which emphasizes contextual information capturing and multi-scale feature fusion. First, CM-Net employs an U-shaped architecture with a Pyramid Vision Transformer (PVT) as the encoder which can effectively model global information and extract multi-scale features. Second, we design a Spatial Channel Self-Attention (SC) module which can capture both global and local long-range contextual information, and suppress image noise, thereby enhancing local details based on well-modeled global features. Furthermore, we propose a Multi-Scale Feature Fusion (MSF) module, which is designed to preserve low-level spatial details and high-level semantic features during the fusion process. Finally, extensive experiments on six publicly available polyp datasets demonstrate that our method outperforms widely adopted polyp segmentation models for colonoscopy. The source code of the propose model CM-Net will be publicly available at https://github.com/Icandonothing/CM-Net .