Skeleton data consists of complex spatial-temporal relationships. LSTM-based methods have been widely used to capture temporal features but face challenges in processing spatial features due to their sensitivity to input order. Moreover, existing positional encodings struggle to preserve skeletal topology. To address these issues, we propose 3SmLSTM, a novel LSTM-based approach, with Specific skeletal connectivity encoding, Symmetric feature mixing MatrixLSTM (mLSTM), and spatial-temporal Second-order pooling. The encoding introduces sequentiality to the joint sequence, enabling mLSTM to better process spatial information while preserving the skeletal topology. The feature mixing mechanism extracts multi-scale temporal information and compensates for non-sequential constraints, while the spatial-temporal second-order pooling captures correlations across channels of different action details for enhancing generalization. Experiments on NTU RGB+D and NTU RGB+D 120 demonstrate the superior performance of 3SmLSTM compared to the state-of-the-art methods. The code is available at https://github.com/StarPlatinumDa/3SmLSTM .