Glaucoma is one of the major causes of blindness in the world that affects millions of people. Early detection and treatment are essential to prevent vision loss. This research introduces a novel hybrid model, namely 7L-DCNN-GBC, to classify the glaucoma severity stage. This model combines a 7-layer Deep Convolutional Neural Network (7L-DCNN) with a Gradient Boosting Classifier (GBC) where 7L-DCNN extracts the feature and GBC classifies the severity level. The regions’ YOLOv8-based instance segmentation is used for more accurate classification and localization. In this research, four datasets were considered for model performance evaluation; three benchmark datasets, namely REFUGE, ORIGA, and DRISHTI-GS) and one newly proposed dataset, namely AKMC, were gathered from a local hospital, Anwer Khan Modern Medical College Hospital (AKMMCH) in Dhaka, Bangladesh. The dataset contains 256 high-quality fundus images that were annotated by a retinal specialist team. The accuracy, precision, recall and specificity metrics are considered for model performance evaluation. The proposed dataset obtained the highest result with an accuracy, f1-score, precision, recall, roc, and specificity of 99.13%, 98.71%, 98.54%, 98.89%, 98.68%, and 98.79%, respectively. In addition, the YOLOv8-based instance segmentation model impressively identifies the region of interest with 98.71% mAP50 and 94.4% recall. The proposed model is notable for clinical application, aiding early glaucoma detection and personalized treatment strategies.