Cardiovascular diseases (CVDs) are the leading cause of death worldwide, highlighting the need for precise diagnostic and therapeutic strategies. Whole heart segmentation (WHS) from medical images is vital for understanding cardiac anatomy, assessing disease progression, and guiding treatments. However, WHS poses significant challenges due to the dynamic shape changes of the heart during the cardiac cycle, motion artifacts, low contrast-to-noise ratio, and variability from multi-center data. Additionally, CT and MRI modalities each present unique difficulties: CT provides high spatial resolution but poor soft-tissue contrast, while MRI offers excellent soft-tissue contrast but is prone to artifacts. To address these challenges, we propose a two-stage method. In the first stage, the DINOv2 framework leverages a 3D Vision-LSTM (xLSTM) model for self-supervised learning on the limited, unlabeled WHS++ dataset to overcome the scarcity of annotated data. In the second stage, we fine-tune the model for segmentation by freezing the encoder and optimizing the decoder. Our approach is rigorously evaluated on the WHS++ dataset for both CT and MRI modalities, demonstrating superior performance compared to state-of-the-art deep learning models in accurately segmenting cardiac structures. Our proposed solution achieved an average Dice Coefficient of 0.9414 using CT and 0.8891 using MRI dataset. This work advances the field of cardiac imaging by providing a robust, generalizable solution for automated whole heart segmentation.