<p>Colonoscopy is the gold standard for the examination and detection of polyps, with over 90% of polyps potentially progressing into colorectal cancer. Accurate polyp segmentation plays a pivotal role in the early diagnosis and treatment of colorectal cancer. Most existing methods ignore inconsistent colors of colonoscopy images and fuse features of different layers by directly using element-wise addition or concatenation. The former operation may lead to overfitting, while the latter can weaken the complementarity between different levels. In order to resolve these challenges, we propose a deep adjacent-differential network with shallow attention for polyp segmentation (ADSANet). Initially, we develop a color exchange strategy based on uncorrelated and specular region suppression to decouple image content from colors. This strategy allows the model to prioritize the target appearance, reducing the risk of overfitting to color features. To maximize the synergy between layers, we first propose an adjacent-differential feature fusion module (ADFM) and then employ the shallow attention module (SAM) for further feature fusion. Specifically, the ADFM generates differential features between adjacent layers and combines features at the corresponding level in the encoder, as well as the adjacent decoder features. We sequentially apply the ADFM at different scale levels for feature decoding, and the final prediction is computed by fusing the outputs of the sequentially connected ADFMs and the shallow attention module. Extensive experiments on five datasets show ADSANet outperforms most state-of-the-art convolutional neural networks (CNN)-based methods, ADSANet yields significant gains of 18.5%, 3.5%, 3.8%, 4.0%, and 1.7% over classical method PraNet on ETIS, ClinicDB, Endoscene, ColonDB, and Kvasir-SEG respectively, which demonstrates the effectiveness of the proposed scheme of color exchange and adjacent-differential feature fusion for more accurate polyp segmentation.</p>