Medical image segmentation plays a crucial role in quantitative analysis, clinical diagnosis and medical interventions. However, extracting valuable information from medical images is challenging due to the presence of different object types and scales, complex backgrounds and tissue similarities. To address these issues, LANet, a Lightweight Attention Network which incorporates an Efficient Fusion Attention (EFA) block and an Adaptive Feature Fusion (AFF) decoding block was presented in the paper. The EFA block enhances the model’s feature extraction capability by capturing task-relevant information while reducing redundancy in channel and spatial locations. The AFF decoding block fuses the purified low-level features from the encoder with the sampled features from the decoder, enhancing the network’s understanding and expression of input features. Additionally, the model adopts MobileViT as a lightweight backbone network with a small number of parameters, facilitating easy training and faster predictive inference. The efficiency of LANet was evaluated using four public datasets: kvasir-SEG, CVC-clinicDB, CVC-colonDB, and the Data Science Bowl 2018. Experimental results demonstrate that LANet outperforms state-of-the-art methods in terms of intersection over union (mIoU) and Dice scores.