The semantic segmentation of 3D scenes, particularly in indoor environments, Indoor point cloud semantic segmentation remains a pivotal challenge in 3D vision. While Transformer-based methods outperform traditional CNNs, their limited geometric modeling capabilities hinder effective representation of complex local structures. To address this, we propose GAPFormer, a geometry-adaptive Transformer that integrates a spherical harmonic geometric feature modulator for direction-aware geometric encoding, a geometrically-aware dynamic positional bias module to enhance local perception, and a point-wise convolution-based attention module for multi-level feature fusion. Experiments demonstrate state-of-the-art performance on ScanNet v2 (77.35%, +0.23%), S3DIS (72.6%, +0.8%), and ScanNet200 (36.06%, +0.86%) over Point Transformer v3, with notable gains in small-object segmentation, offering a robust solution for precise 3D scene parsing.