While Transformer-based models like SegFormer have set new standards in semantic segmentation, they exhibit critical limitations in high-resolution urban remote sensing. Specifically, SegFormer’s inherent weak locality bias leads to blurred object boundaries, and its coarse deep-feature representations struggle to detect small-scale objects, compounding the challenge of high computational costs for real-time processing. To overcome these shortcomings, we propose GCS-SegFormer, an enhanced architecture that surgically improves segmentation accuracy and efficiency. GCS-SegFormer introduces three targeted enhancements: (1) a lightweight Global Channel Spatial Attention (GCSA) module to refine deep features, significantly improving sensitivity to small objects; (2) a Semantics and Detail Infusion (SDI) module that creates a powerful, semantically-aware fusion pathway to reconstruct sharp object boundaries; and (3) the strategic integration of Ghost convolutions in the decoder to drastically reduce computational complexity. Extensive experiments on the ISPRS Vaihingen and UAVid datasets show that GCS-SegFormer achieves state-of-the-art mIoU scores of 93.25% and 75.49%, respectively, outperforming the baseline SegFormer by 1.14% and 0.73% while reducing computational complexity by approximately 33.16%.