Medical image segmentation is crucial in medical image analysis. In recent years, deep learning, particularly convolutional neural networks (CNNs) and Transformer models, has significantly advanced this field. To fully leverage the abilities of CNNs and Transformers in extracting local and global information, we propose HSINet, which employs Swin Transformer and the newly introduced Deep Dense Feature Extraction (DFE) block to construct dual encoders. A Swin Transformer and DFE Encoded Feature Fusion (TDEF) module is designed to merge features from the two branches, and the Multi-Scale Semantic Fusion (MSSF) module further promotes the full utilization of low-level and high-level features from the encoders. We evaluated the proposed network on the familial cerebral cavernous malformations private dataset (SG-FCCM) and the ISIC-2017 challenge dataset. The experimental results indicate that the proposed HSINet outperforms several other advanced segmentation methods, demonstrating its superiority in medical image segmentation.