Gait recognition has emerged as a promising biometric identification technique due to its non-invasive nature and distance operability. However, existing approaches struggle with robustness against scenarios across different actions with varying walking patterns. To address this issue, we study action-agnostic gait recognition based on human skeletons. We propose a Multi-Granularity Spatio-Temporal Mixer (MG-STM), a novel pose-based gait recognition framework that hierarchically partitions skeletal data into joint, part, and body-level representations to capture multi-granularity motion patterns. The encoder integrates self-attention mechanisms with large-kernel convolutions to learn action-invariant spatio-temporal features at different hierarchical levels. Experimental results on NTU RGB+D and CASIA-B demonstrate substantial improvements of our approach over the SOTA methods, and the ability to identify the person based on gait across different actions.