The foreign object debris (FODs) on airport runways often disrupt operations while taking off and landing flights, leading to accidents. Small-scale FODs cannot manually rule out on time, which threatens aviation safety. This paper presents an intelligent computer vision system for small-scale FOD detection. This work proposes a Feature-Fusion Yolo (FF-Yolo) to accelerate the Yolov5 model to detect FODs in airports. A lightweight convolution-based attention module (CBAM) is considered in the backbone of the proposed architecture to improve the model efficiency by focusing on the target features. In addition, to reduce the overfitting problem, a C3TR module is included in the FF-Yolo model’s backbone and neck, which captures both spatial and temporal features. Further, GhostConv is used in neck network, which helps in increasing the overall accuracy. Finally, an improved detection head is introduced in FF-Yolo to find out the size of the small-scale FODs along with their pixel location, which helps the aviation personnel to measure the severity and take prompt action. The experiments are performed on a FOD-A dataset with a runway and taxiway background, including different light and weather conditions. The proposed model achieved 98.61% mAP@0.5 and 83.21% mAP@0.95, which are higher than other state-of-the-art (SOTA) models. An overall improvement of 7.33%, 6.43%, 5.79%, and 4.93% of mAP@0.5 and 6.26%, 16.51%, and 5.1% of mAP@0.95 are noted compared to the Yolov5 baseline models. The FF-Yolo model achieves a detection speed of 33.31 frame/s, much higher than the other SOTA models. Consequently, the ablation study verifies the robustness of the FF-Yolo model for small-scale FOD detection.