Colon cancer is among the leading causes of cancer-related death worldwide for both men and women, with colorectal polyps serving as a significant predisposing factor. Early polyp identification and removal-the precursors to colorectal cancer-is essential to its prevention. Colonoscopy is considered the gold standard for colorectal cancer screening because it allows for the immediate removal of polyps, preventing them from developing into cancer. Despite its effectiveness, conventional colonoscopy is time-consuming, highly labor-intensive, and prone to human mistakes. Therefore, we modified the efficient object detection model, YOLO-V8, to develop our novel approach, EDF-YOLO8, for automating polyp identification. Our model employs deformable convolution in the bottleneck as a robust solution for effectively detecting polyps of various sizes. We enhance the effectiveness of our model by incorporating the Exponential Linear Unit (ELU), which further increases the detection accuracy and tends to accelerate the model learning process. We trained and tested the suggested model on two distinct datasets from publicly accessible sources and conducted thorough assessments to ensure its robustness and generalizability. The proposed model achieved an outstanding performance, attaining a mAP50 score of 0.931 and 0.894 for the Kvasir and Polypgen datasets, respectively. Performance analysis demonstrates the efficiency and robustness of our model in accurately detecting polyps from colonoscopic frames from different datasets.