X-ray security inspection is essential for ensuring public safety and protecting lives and properties.However, current deep learning methods for X-ray detection on edge devices often grapple with poor real-time performance, low accuracy and high false alarm rates. To address these challenges, we propose YOLO-Xray, a novel algorithm that builds on an existing detection framework. We enhance the architecture by introducing the ContextFusion module, which integrates contextual semantic information to improve the interpretation of complex X-ray scenes, and develop two specialized loss functions, EdgeOBBLoss and BackgroundSensitiveLoss, designed specifically for X-ray imaging to enhance small object detection and the differentiation of overlapping targets.Experimental results on the PIDray and OPIXray datasets demonstrate that YOLO-Xray outperforms previous state-of-the-art methods, achieving a 12.3% improvement in mAP50–95 on PIDray and a 5.9% improvement in mAP50 on OPIXray, while maintaining high speed with 160 FPS on PIDray and 150 FPS on OPIXray. These enhancements ensure both superior accuracy and rapid real-time detection on edge devices, making YOLO-Xray an efficient and robust solution for X-ray security inspection.