Concrete crack detection is a critical task in structural health monitoring, facing challenges such as low contrast, scale variations, and contextual information. While previous studies have proposed advanced vision-based methods using CNN architectures, the persistence of errors and wrong predictions necessitates innovative approaches. In this study, we propose an ensemble approach that not only accurately detects cracks but also classifies their severity as major or minor. Our method leverages transfer learning, combining pre-trained models, and fine-tuning their predictions with optimized weights. Through ensemble learning, models learn to complement each other’s weaknesses and limitations, resulting in more accurate predictions across various scenarios. We evaluate our approach using metrics such as Intersection over Union (IoU), recall, and F1 score, and test it on real crack data. The proposed ensemble method achieves an IoU score of 0.86 and an accuracy of 0.9944, confirming its consistent accuracy and robustness. This ensemble approach minimizes mistakes, ensuring a robust and reliable concrete crack detection system for structural health monitoring applications.