Unsupervised Visible-Infrared Person Re-Identification (USL-VI-ReID) focuses on addressing the challenge of matching individuals across different modalities. Most previous USL-VI-ReID works aim to eliminate modality discrepancies and achieving cluster-level alignment to facilitate cross-modality matching. However, they often overlook two critical aspects: taking fine-grained instance-level relationships into account, and learning modality-invariant features together with modality-specific features simultaneously, which can significantly affect the re-identification accuracy. To resolve these above challenges, we introduce an Affinity-Guided Multi-Grained Label Calibration (AMGLC) module to mine the fine-grained instance-level relationships from pedestrian images in both modalities to guide noisy-label calibration, coupled with an Affinity-Weighted Multi-Memory Learning (AMML) module, which fully exploits the modality-invariant as well as modality-specific features of pedestrians. Systematic experimentation on reference datasets substantiates the superiority of our method.