The currently existing gait recognition methods mainly focus on detecting walking patterns based on synchronized data, especially the appearance-based methods using GEIs as original inputs. These methods need large amount of work to deal with data preprocessing to generate a fine-grained gait cycle for each subject, which is an obstacle to use gait recognition widely in real-word applications. To solve this problem, we propose a model-based gait recognition method that is able to generate gait representations from unsynchronized data and consider the dependencies between joints simultaneously. Firstly, we utilize simple joint-wise autoencoder groups to reconstruct unsynchronized joint sequences generated by pose estimation, where multi-scale time patches are employed to capture time-series patterns by capturing information from variable length windows. Then, to calculate the inter-joint dependencies, a feature fusion structure based on self-attention mechanism is used to generate final gait representations considering all joints’ moving patterns. This pipeline gives out temporal-spatial gait representations from unsynchronized data. Experimental results demonstrate that the performance can be comparable to the state-of-the-art synchronized pose-based gait recognition methods. Moreover, extensive ablation studies also explain the effectiveness of the proposed method’s structure design.