Micro-expressions (MEs) are subtle, involuntary facial muscle movements that occur without conscious awareness. Due to their fine-grained local variations and significant individual differences, micro-expression recognition (MER) remains a challenging task. To address this, we propose a novel recognition network (PGS-Net). First, we design a Self-Supervised Displacement Reconstruction Module (SDRM) that reconstructs and amplifies the displacement fields between the onset and apex frames, refining local motion information and enhancing the modeling of ME details. Then, we introduce a Displacement-Guided Personalized Graph Structure Module (DPGSM), which combines facial structure and displacement features to construct dynamic personalized graph topologies, adapting to differences in individual facial structures and motion patterns. Experiments on three public datasets—SMIC, CASME II, and SAMM—show that PGS-Net outperforms existing methods in three-class classification tasks in terms of UF1 and UAR, demonstrating strong recognition performance and generalization capability.