Visible-Infrared Person Re-identification (VI-ReID) would effectively improve the recognition performance in weak-lighting and nighttime scenes, which is an important research direction in pattern recognition and computer vision. However, existing methods usually focus on reducing the image differences between modalities (visible and infrared) to extract more reliable features, while neglecting the ability to discriminate the different identities with similar appearances. To address this problem, we propose a framework called “Progressive Diversity Expansion Transformer (PDET)”, which includes a Diversity Distinguishing Vision Transformer Module (DDViTM) and a Cross-Modality Similarity Matching (CMSM) module for VI-ReID in this study. The DDViTM is proposed to implement the multiple embedded output vectors for a single input, learning feature representations of individual pedestrians in different modalities. The second module (CMSM) is used to improve the feature similarity between visible and infrared images, and dynamically adjust the image sequence weights of the two modalities to complete the training and optimization efficiency for the entire network. We conducted extensive experiments on the SYSU-MM01 and RegDB datasets, widely recognized public datasets for VR-ReID. The results demonstrate that the algorithm presented in this work has achieved promising performance compared to state-of-the-art methods. The code is available at  https://github.com/jxsiaj/PEDT.git .