Early detection and classification of breast tumors via mammographic images are critical for reducing mortality rates. We introduce a lightweight and efficient tumor detection model built on an enhanced YOLOv8 framework to support diagnosis. Firstly, we introduce a multi-scale lightweight convolution module named MSLConv. We further integrate batch normalization and the SILU activation function to create a comprehensive convolutional module called MSLCBS. To further enhance the model, we have restructured the detection head to substantially decrease the parameter count in this segment of the network. Additionally, to capture subtle features such as microcalcifications and irregular textures, we incorporate Focal Modulation to improve feature fusion capabilities. This approach aggregates and interacts with contextual information in a lightweight manner. Considering the diversity in the shape, texture, and size of breast tumors, we introduce WISE-IOU, a bounding box regression scheme to enhance the model’s generalization capability. All of the extensive experiments on CBIS-DDSM and INBREAST datasets have proven that our method achieves the highest mAP50 score with the fewest parameters and computational load, which validate its reliability.