In the problem of crowd counting, accurately estimating the number of people in complex scenarios remains a challenging task. In this paper, we propose GramFormer-based architecture with a learnable Fourier encoding and sliding window attention mechanism for crowd counting. It aims to improve the flexibility of spatial position encoding and the diversity of input features. Specifically, we integrate learnable Fourier features into Gramformer’s embedding layer for multi-dimensional spatial position encoding. This approach allows the model to learn optimal frequency parameters in a data-driven manner, reducing the need for manual frequency and scale adjustments. Additionally, the input feature map passes through a sliding window attention module, which captures local variations in the density of people, retains local details, and integrates global context. Finally, local features are fused with the global learnable Fourier features to enhance the input to the GramFormer. To demonstrate the superiority of the proposed method, the performance comparison between our method and state-of-the-art methods is conducted on four crowd counting databases. The results demonstrate that our method outperforms competing methods in terms of MSE and MAE.