Accurate medical image segmentation is essential for physicians to obtain high-quality diagnostic results. While recent hybrid Transformer and CNN based methods have significantly improved the performance of biomedical image segmentation to meet the precision requirements of medical imaging, they still face the following challenges: 1) difficulties in effectively reducing the semantic gap between encoder and decoder features due to simple feature concatenation or addition strategies. 2) challenges in efficiently fusing spatial and semantic information among cross-scale features during the decoding process while preventing redundant information output. To solve the above problems, this paper proposes a novel medical image segmentation method based on the cross attention mechanism and refined feature fusion strategy (CRFNet). The proposed approach utilizes higher-order spatial interaction block based on recursive gated convolution (HorBlock) as the encoder to capture global range dependencies and provide useful induced biases for the segmentation task. In the decoding process, we design the channel spatial cross attention module (CSCA) and coarse to fine feature fusion module (C2 \(F^3\) ). CSCA effectively reduces the semantic gap between the encoder and decoder by facilitating the cross-fusion of encoder and decoder features in both channel and spatial dimensions. C2 \(F^3\) enables efficient collaboration between cross-scale contextual features in the decoder to obtain a more refined segmentation result. Extensive experiments in five datasets demonstrate that our proposed model outperforms state-of-the-art methods in the segmentation of skin lesion images, breast ultrasound images and polyps in colonoscopy images.