<p>Diabetic Retinopathy (DR) and Diabetic Macular Edema (DME) are two of the most severe complications of diabetes, often leading to irreversible vision loss. This paper introduces the Attention-Driven Retinal Classification Model (ADRCM), a novel lightweight deep learning architecture explicitly designed for the joint grading of DR and DME, a challenge inadequately addressed in existing research. The core objective is to improve grading accuracy while maintaining model generalizability across datasets. ADRCM follows the Hierarchical Attention-Fusion Network (HAFNet) architecture. HAFNet module is composed of two crucial elements: the Feature Fusion and Enhancement Module (FFEM) and the Spatial Self-Attention Block (SSAB). The FFEM extracts and enhances salient features from the retinal fundus images, effectively handling variations in lesion sizes and appearances, by fusion of convolutional operations with varying receptive fields and dilation rates. At the same time, SSAB utilizes spatial self-attention mechanisms that allow for the selective enhancement of feature maps, focusing on specific regions of interest relevant to the accurate grading of DR and DME. This model which operates on an enhanced spatial self attention mechanism with multi-scale feature extraction and fusion strategy, ensures comprehensive analysis of the retinal images. A novel learning strategy named Enhanced Cross-Validation Weight Integration (ECVWI) is introduced in this paper, achieving a highly generalized model capable of handling diverse datasets. The proposed ADRCM achieved accuracy, precision and recall values 0.9948, 0.9953 and 0.9942 respectively on the Messidor dataset. It also correctly classified all test samples in the IDRiD dataset, significantly outperforming existing methods in joint DR and DME grading tasks. This lightweight solution offers a significant step forward in automated retinal disease grading, offering potential for enhanced screening and early diagnosis in clinical settings.</p>