<p>Colorectal cancer (CRC) is one of the most common sources of cancer-related death worldwide. Early detection of these precancerous polyps with the aid of colonoscopy plays an important role in decreasing the burden of CRC. By employing novel optimization techniques, this work proposes a new deep learning-based approach to automate polyp detection using colonoscopy images. Specifically, we use the CaffeNet architecture for extracting features and a Support Vector Machine (SVM) for classification. With the goal of improving both stages, the Refined Single Candidate Optimizer (RSCO) is presented to eliminate the imperfection of traditional optimization approaches and refine the search mechanism from the insight of particle swarm optimization (PSO). This approach demonstrates great potential for supporting dynamic equilibrium between exploration and exploitation in the optimization process, which further helps to improve feature extraction and classification performance. The performance of our proposed model is evaluated on the SUN Colonoscopy Video Database, and its effectiveness is compared with the conventional methods including CNN/SVM, DNN, GAN2, and Dual-path convolutional neural network (DP-CNN). We show better performance in terms of precision, recall and accuracy, and provide evidence for the efficacy of the proposed approach for early polyp detection on routine colonoscopy to assist in the timely diagnosis of CRC.</p>