Atrial fibrillation (AF), a widespread cardiac arrhythmia, is associated with significant morbidity and mortality. However, current treatments are limited due to an incomplete understanding of the atrial anatomy that drives AF. The MBAS 2024 Challenge builds on our 2018 left atrium challenge by expanding the focus to include both the left and right atria. Leveraging the largest bi-atrial LGE-MRI dataset to date, this challenge centers on advanced segmentation and biomarker identification to enhance treatment strategies for AF. The goal of this challenge is to develop automated medical semantic segmentation methods using a specialized 3-class bi-atrial dataset, ultimately aiming to improve the performance of AF treatments. We have introduced a novel two-stage process for the precise segmentation of the atrial walls and both atria. Our approach begins by utilizing self-supervised learning techniques tailored for 3D medical imaging, enabling the model to effectively capture the complex anatomical structures present in 3D medical datasets. Following this pre-training stage, the learned knowledge is transferred to a targeted 3D medical image segmentation task. In the second stage, we present an xLSTM-based model enhanced with the integration of Self-Supervised Learning (SSL) and cross-window attention mechanisms. This innovative approach has demonstrated superior performance in segmenting the atrial walls as well as the left and right blood pool cavities, offering a promising advancement in the accuracy and effectiveness of AF treatment through more detailed anatomical segmentation. Our approach achieved overall second ranked on validation leaderboard.