Accurate breast tumor segmentation and malignancy detection are crucial for early cancer diagnosis. In this context, we propose a novel lightweight multi-task learning framework, MA-DTNet, designed to perform both tasks simultaneously in an encoder-shared scenario. This approach leverages shared representations and contextual information, enabling mutual enhancement of the tasks. Unlike existing methods that require a large number of trainable parameters, MA-DTNet integrates a Spatial Morphological Attention (SMA) module alongside a Channel Attention (CA) mechanism to strategically enhance crucial morphological features and emphasize informative channels within the extracted representations. The SMA mechanism combines traditional morphological operations with trainable, adaptive structuring elements, effectively highlighting critical morphological attributes of regions of interest (ROIs) of various shapes and sizes within medical images. This targeted emphasis on morphological features translates to improved performance in both segmentation and classification tasks. Notably, MA-DTNet demonstrates superior performance compared to state-of-the-art multi-task learning (MTL) and single-task models on two publicly available breast ultrasound datasets. Specifically, on the UDIAT dataset, our approach achieves a 3.28 \(\%\) and 1.05 \(\%\) enhancement in dice score (segmentation) and F1 score (classification), respectively. Similarly, for the BUSI dataset, MA-DTNet exhibits a 1.62 \(\%\) and 4.96 \(\%\) improvement in dice score and accuracy, respectively. Significantly, MA-DTNet achieves these performance gains with significantly fewer trainable parameters than existing methods, underscoring its efficiency and potential for real-world applications. The method’s generalization ability is further tested on two additional multi-task learning tasks: segmenting and classifying glands in histology images and segmenting and classifying skin lesions in dermoscopic images.