The YOLO series of algorithms has become the primary method for real-time object detection. Many studies have enhanced the benchmark performance by adjusting model structures, updating training methods, and optimizing hyperparameters. However, in underground coal mine environments, factors such as dim lighting and complex backgrounds lead to small targets being easily confused with the background or occluded, significantly reducing detection performance. This paper proposes the MineTinyNet-YOLO, which is suitable for small target detection in complex underground coal mine scenarios. Firstly, we introduce the Dynamic Boundary-Aware Convolutional Layer (DBACL) to dynamically adjust the receptive field for target detection, suppress background interference, and enhance the detection capability for occluded targets. Through the Dynamic Smoothing Convolution (DSC), we effectively smooth boundary artifacts, capturing the shape and boundary information of targets more accurately. Secondly, the Adaptive Interactive Feature Network (AIF-Net) is designed to efficiently fuse cross-layer information, alleviating the issues of slow fusion speed and spatial information loss that are inherent in the CSPSPP layer. The Point-wise Upsampling Operator (PUO) is introduced to reduce computational costs and restore more detailed information. Extensive experiments conducted on self-built and public datasets demonstrate that the proposed algorithm achieves an mAP of 91.1%, representing a 2.4% improvement over the YOLOv7-tiny algorithm, thereby validating the effectiveness of our approach. Additionally, our method boasts a parameter count of 6M and an inference time of 0.329 s, a reduction of 0.173 s compared to the YOLOv7-tiny algorithm, making it more suitable for deployment on mobile devices.