Medical image segmentation is one of the main steps in enhancing diagnostic accuracy, especially in colorectal polyp detection. Classic encoder–decoder architectures, such as U-Net and its variants, have dominated this by integrating spatial and semantic features. However, they cannot model global dependencies because of their structure, seriously limiting their effectiveness in complex tasks. In this work, we compare U-Net, Attention U-Net, ResUNet++, and the transformer-based UNETR to evaluate the evolution of segmentation models. Experiments on the Kvasir-SEG and CVC-ClinicDB datasets show that U-Net and Attention U-Net achieve the best performance, with dice similarity coefficients of 85.98% and 90.60% on CVC-ClinicDB, respectively. UNETR reported lower accuracy, 86.84% DSC, but with significantly fewer parameters, 2.5M compared to the over 30M of U-Net, which is efficient for resource-constrained applications. These results indicate a promising future for transformer-based architectures in efficient medical image segmentation. Further development in this area should be geared toward improving their accuracy with sophisticated attention mechanisms and hybrid designs for a wider and successful clinical adoption by exploiting better computational advantages.