As road infrastructure plays a critical role in modern transportation, ensuring road surface safety and durability is paramount. Identifying potholes on roadway surfaces presents a significant obstacle in road infrastructure, affecting motorists’ structural strength and well-being. This paper presents a novel approach to road pothole classification by leveraging the strengths of Convolutional Neural Networks (CNN) and Extreme Learning Machines (ELM). The proposed model integrates CNN with ELM for feature extraction and with rapid and efficient hierarchical learning capabilities. The collaborative effort between CNN and ELM improves the pothole classification’s overall performance and effectively handles concerns related to computational time. Experimental results on real-world road surface datasets demonstrate the effectiveness and robustness of the proposed C-ELM approach. Integrating deep learning and ensemble methods offers a promising solution for advancing the state-of-the-art in intelligent transportation systems, contributing to enhanced road safety and infrastructure maintenance.