Medical image analysis often relies on models optimized for specific tasks, such as classification or detection. However, this single-task approach limits the utilization of shared features across tasks and becomes particularly inefficient when data availability is limited. In this paper, we investigate the potential of multi-task learning (MTL) with Visual Transformers to optimize both classification and reconstruction tasks, especially when training data is scarce. Using datasets like BRATS (binary classification of brain tumor presence on MRI slices) and ultrasound images of muscles (for identifying pathologies such as Myopathy, Myelopathy, and Polyneuropathy), we evaluate MTL against standalone and pre-training paradigms. Results indicate that MTL significantly enhances model performance, particularly in classification tasks, by leveraging shared representations and improving attention mechanisms. Our findings demonstrate that MTL mitigates the challenges of limited data availability by effectively transferring knowledge between tasks, making it a valuable strategy for medical imaging applications.