For medical image segmentation, transformer-based models have exhibited superior segmentation performance. However, their high computational complexity continues to pose a major challenge. In contrast, Mamba offers a more computationally efficient alternative; however, its performance remains inferior to that of transformers. This study proposes a U-Net-based novel lightweight hybrid model, SwiM-UNet, which is the first Mamba transformer hybrid model specifically designed to process 3D data. Specifically, efficient TSMamba (eTSMamba) is utilized in the initial stages of the U-Net architecture to efficiently manage computational overhead, while efficient Swin transformers (eSwin) are employed in the later stages to effectively capture long-range dependencies and local contextual information. In addition, this model strategically integrates both the Mamba and Swin transformer architectures using a Mamba–Swin adapter (MS-adapter) to leverage their complementary advantages. The proposed MS-adapter consists of three sub-adapters that play different roles in emphasizing local information between eTSMamba and eSwin modules and gates that balance the sub-adapters. This model also employs a low-rank MLP in the encoder and applies channel reduction in the decoder to enhance computational efficiency. We conducted performance evaluations using the publicly available BraTS2023 dataset and confirmed that the proposed model outperformed state-of-the-art benchmark models while significantly reducing computational complexity.