In medical image segmentation, existing methods fail to deal with the long-range correlations, imbalances between foreground and background, and indistinct tumor edges. In order to overcome these problems, we present a novel U-shaped deep network with novel encoding modules that can preserve critical features for tumor segmentation tasks. Additionally, Bi-level routing attention module (named Biformer)is designed with robust attention mechanisms and sparse patterns to address the imbalance between the foreground and background in ultrasound images. Moreover, based on an important observation that high-frequency components can be served as visual prompts to improve segmentation precision, especially in delineating tumor edges, the proposed network, named Biformer U-Net (BF-UNet), integrates the Biformer module and the high-frequency component visual prompt module into the U-shaped architecture. The proposed method was tested on four datasets, and the results show that BF-UNet performs better than several other segmentation algorithms.