This paper presents a Fine-grained Feature Extraction U-Net (FgFEU-Net) model to realize automatic localization and segmentation of breast cancer lesions with breast ultrasound images. In the FgFEU-Net model, we used the transfer learning method to replace the coding part of the backbone network with the VGG16 model to achieve deep and fine-grained feature extraction. The extracted low-level information is combined with the high-level knowledge of the U-Net decoding layer at the same layer and converted into high-resolution information to pinpoint lesions precisely. Finally, the high-resolution information is transformed into a high-resolution image through the final convolutional layer. In addition, the combo loss is adopted to deal with the imbalance between organ input images and their output images. The proposed model was tested and predicted in the BUS2018 (Breast Ultrasound Images 2018) dataset, and the proposed evaluation metric of accuracy, precision, and sensitivity reached 99.03%, 96.83%, and 99.35%, respectively.