Visible-infrared person re-identification (VI-ReID) is crucial for surveillance and security applications. Several studies have been performed for supervised VI-ReID, and recent methods show excellent retrieval performance on public datasets. However, obtaining VI-ReID data in practical scenarios presents significant challenges due to the necessity for the same identity to be available across different types of cameras, potentially spanning various locations and time frames, along with the arduous task of annotating data owing to modality discrepancies. This motivates us to explore methods requiring limited data from a select number of identities, which is more readily obtainable. To this end, we introduce a novel two-stage learning framework for VI-ReID that efficiently works with scarce data and labels. Our framework focuses on Supervised Domain Adaptation, where a pre-trained model from a source dataset is utilized on a small annotated target dataset. Additionally, we introduce a novel loss, Hetero-Dissimilarity based Maximum Mean Discrepancy (HD-MMD), tailored for adapting heterogeneous source and target domains. Our approach addresses the inherent challenges of domain shift between datasets and modality differences between visible and infrared imagery. Our proposed method outperforms several label-efficient approaches on public VI-ReID datasets while utilizing significantly smaller amount of data. Ablation analysis conducted with several popular baselines reveals the efficacy of our proposed SDA framework and HD-MMD loss in improving retrieval performance. We also demonstrate the ease of integrating our approach with other methods. Code will be released at https://github.com/Mihirsahu2307/SDA-VI-ReID .