Transformer’s self-attention effectively models long-range dependencies and captures global representations, establishing Transformer-based approaches as a leading method in medical image research. However, in contrast to CNNs, Transformers often struggle to capture local spatial details, potentially resulting in losing fine-grained spatial features and compromising image boundaries or small object detection. To deal with the question, we build a network, Adaptive Boundary Aggregation Network (ABANet), designed to integrate local features with global semantic information, thereby enhancing semantic segmentation quality in diverse medical images. ABANet utilizes a Multi-scope Aggregation Block (MAB) to fuse global and local spatial features, capturing intricate spatial relationships. Furthermore, the Adaptive Boundary Aggregation Decoder (ABAD) aggregates boundary features with semantic information, efficiently combining low- and high-level information. Experiments on medical image segmentation indicates ABANet is helpful and achieves excellent performance.