Medical image segmentation technology urgently needs a lightweight model to meet the requirement for real-time processing, given the increasing expansion of mobile medical devices. This paper addresses the issues of the existing Transformer-based architecture’s high computational complexity and large number of parameters by proposing APG-UNet, a lightweight and effective medical image segmentation network that combines the hierarchical multiscale feature fusion module (AssemFormer), the Group Aggregation Bridge with DAM Module (GABD), and the Prompt Generation Module (PGM) through the design of a lightweight architecture: (1) AssemFormer uses a hierarchical feature extraction mechanism that synchronously captures both the local details and the global context, preventing the loss of information across the scales; (2) GABD achieves cross-scale integration through hierarchical fusion of textural cues, semantic abstractions, and decoder-guided attention masks, significantly reducing computational overhead; (3) PGM adaptively generates cue vectors based on input features, strengthening the ability to characterize the lesion area. Loss of information. According to experiments, APG-UNet decreases the number of TransFuse parameters by 286 times and the computational cost by 128 times when compared to TransFuse, achieving mIoU of 83.78%, 79.69% and 81.26% on the ISIC2016/2017/2018 datasets, respectively. This work offers a lightweight, high-precision, low-latency solution for situations with limited resources. The code is open source ( https://github.com/2193304492zhy/APG-UNet ).