<p>Medical image segmentation is critical for precise diagnostics and therapeutic planning, yet existing methods face challenges in balancing computational efficiency and accuracy, particularly for small-scale targets and ambiguous boundaries. While hierarchical architectures like CNNs and Transformers have advanced the field, they often suffer from limited receptive fields, high computational costs, or inadequate feature fusion. To address these limitations, we propose CSAFD, a novel robust cross-scale attention fusion decoder, which integrates a cross-scale attention fusion module (CSAFM) for dynamic multi-scale feature interaction and an edge-guided spatial aggregation gate (EGSAG) for boundary refinement. CSAFM enhances sensitivity to small lesions through interactive channel-wise attention, while EGSAG leverages adaptive edge-aware constraints to improve segmentation smoothness in low-contrast regions. Extensive experiments on Synapse Multi-organ, ACDC, and breast ultrasound datasets demonstrate that CSAFD achieves superior segmentation accuracy compared to state-of-the-art approaches while maintaining competitive computational efficiency.</p>