Unsupervised visible-infrared person re-identification (USL-VI-ReID) presents significant challenges in cross-modality image matching without annotated training data. Recently, pseudo-label-based methods have emerged as a promising approach in USL-VI-ReID. However, existing methods ignore the discrepancy in the feature distribution caused by random data augmentation, resulting in noisy pseudo labels. Worse still, they usually explore the correspondence between the two modalities at modality-specific cluster-level, leading to insufficient cross-correspondences. In response, we propose a Modality-specific and Modality-agnostic Soft Contrastive Learning (MMSCL) method. Specifically, we design a modality-agnostic hybrid memory to initialize modality-shared prototypes to mitigate the impact of the cross-modality gap. In addition, we introduce soft contrastive learning with a mean-teacher framework to constrain the consistency of the feature distribution. Moreover, an Adaptive-weighted Memory Updating module is proposed to tackle the discrepancy distribution problem in the batch training process. Extensive experimental results on two benchmark datasets demonstrate that our proposed method achieves state-of-the-art performance.