Unlike other medical images, polyp images usually contain a lot of noise interference, which reduces the accuracy of polyp segmentation. To solve the problem of polyp images containing a large amount of noise interference, a Multi-stage Noise Suppression Network (MNSNet) that integrates Transformer and CNN is proposed. Firstly, for the problem that low-level polyp features contain a lot of background noise interference, the Polyp Background Noise Suppression (PBNS) module is constructed based on the self-attention to improve the anti-background noise ability of MNSNet in the feature extraction stage, which in turn improves the network’s performance in polyp segmentation. Secondly, to address the lack of anti-interference ability of the semantic fusion method in the existing polyp segmentation network, the Polyp Dynamic Noise Suppression (PDNS) module is constructed based on the dynamic kernel method to improve the adaptability of MNSNet to complex and variable noise interference in the polyp images during the semantic fusion stage, thereby improving the network’s polyp segmentation accuracy. Experiment results show that the MNSNet has best performance compare with five methods (SANet, SSFormer, PPFormer, TransFuse and Meta-Polyp), under five benchmark polyp segmentation datasets (the Kvasir dataset, the CVC-ClinicDB dataset, the CVC-ColonDB dataset, the CVC-T dataset and the ETIS dataset). In particular, compared with the Meta-Polyp, MNSNet improves mDice and mIoU by 2.2% and 2.0% on the ETIS dataset.