Medical image segmentation plays a crucial role in accurate clinical diagnosis, requiring models to capture precise spatial boundary details in high-resolution images. However, during feature fusion, blurred boundaries often lead to boundary shifts and segmentation deviations. In this study, we propose a general medical image segmentation framework, FreqSAM2-UNet. In this framework, we freeze the hiera encoder from the SAM2 model and incorporate adapter fine-tuning to enhance model adaptability. Additionally, we introduce a parallel convolutional branch within the encoder to fully leverage the convolutions contextual capabilities. During feature fusion, we use frequency-aware feature fusion, which uses adaptive high-pass and low-pass filters to effectively improve the transmission of high-frequency features, overcoming problems with blurred boundaries and boundary shifts. Experimental findings demonstrate that FreqSAM2-UNet surpasses state-of-the-art, task-specific methods and achieves superior performance on nine datasets, including polyp and breast cancer segmentation, demonstrating its great potential across different medical image segmentation applications.