Medical image analysis is used extensively in disease detection, diagnosis, and therapy planning at early stages. The study presents a new Multi-Scale Attention-Enhanced Transformer Network (MSAET-Net) for end-to-end fully automated and very accurate medical image classification and segmentation. The framework combines Swin Transformer for global comprehension and U-Net++ for precise spatial localization so that both local fine-grained details and global long-range dependency are captured very effectively. Using contrastive self-supervised learning for improved generalization, the model is trained and evaluated on extensive multimodal datasets, including MRI, CT, and X-ray scans. According to experimental data, MSAET-Net outperforms conventional CNN-based designs by achieving state-of-the-art performance with an average classification accuracy of 98.5%, segmentation Dice score of 94.8%, and sensitivity of 97.2%. These findings show how MSAET-Net has the potential to improve precision medicine and support clinical decision-making.