Accurately extracting building information from high-resolution remote sensing images is of great significance in urban planning, land use management, and related fields. However, challenges such as shadow interference, tree occlusion, and the complex, diverse structures of buildings make fast and accurate building extraction from remote sensing images a highly challenging task. To address this challenge, this paper proposes GASNet, a novel framework designed to enhance feature representation through global spatial dependency modeling and multiscale boundary refinement. First, we introduce the Dual-Scale Dependency Module (DSDM), which leverages graph-based reasoning in non-Euclidean space to dynamically aggregate local and global spatial dependencies while suppressing redundant features. In addition, the Scale-Aware Efficient Attention (SAEA) mechanism is proposed to enhance feature representation along both horizontal and vertical directions, enabling comprehensive boundary information capture. By leveraging the integrity of buildings, it effectively mitigates occlusion-induced interference, significantly improving the accuracy and completeness of building extraction. Extensive experiments on three benchmark datasets—WHU, Massachusetts, and Inria Aerial—demonstrate the superiority of GASNet. Our method achieves state-of-the-art performance, surpassing existing approaches by margins of 1.73% IoU on WHU, 0.48% IoU on Massachusetts, and 1.28% IoU on Inria.