<p>Convolutional Neural Networks (CNNs) are extensively used deep learning architectures because they can optimize feature extraction and classification together. This makes them particularly effective for complex tasks such as medical image analysis. However, CNNs often have difficulty capturing fine and broad features from medical images, resulting in missing crucial areas and reducing their performance. To resolve this challenge, a novel deep learning framework with a model called RobustAttenNet is proposed for medical imaging analysis. The proposed model extracts fine and broad features from medical images through multi-scale feature extraction and fusion by integrating the attention layer with the convolutional layer at different stages. The proposed framework consists of two phases. The first phase focuses on classification performance. In the second phase, robustness is assessed using unseen inputs, adversarial inputs, and noisy images. Although lightweight, with only 0.144 million (M) parameters, significantly fewer than existing models, the model achieves exceptional accuracy of 97.24% on the Brain MRI dataset and 98.67% on the Chest X-ray dataset. Its effectiveness is validated through comparisons with state-of-the-art and pre-trained models, including ResNet-50, VGG-16, and XceptionNet. Furthermore, it maintains strong robustness, exhibiting minimal accuracy deviations (1.24% on Brain MRI and 1.89% on Chest X-ray) under unseen inputs, limited adversarial accuracy drops, and consistently low Bit Error Rate (BER) in the presence of distortions. These combined accuracy, robustness, and efficiency strengths highlight its potential for medical imaging applications.</p>