Detecting prohibited items in X-ray images is crucial for ensuring the safety of transportation hubs. Computer vision algorithms significantly enhance the efficiency of manual inspections. However, X-ray images present unique challenges: semi-transparent overlapping objects, cluttered backgrounds, and deliberately hidden items, which limit the accuracy of common object detection methods. To alleviate these issues, we suggest a High-level Semantic Guidance and Edge Feature Fusion network named HENet for accurate prohibited item detection. Specifically, we design a High-level Semantic Guidance-based Feature Selection (HSGFS) module to filter irrelevant background noise from mid-level features. Then, we create a Multi-scale Edge Feature Fusion Head (MEFFH) that leverages the rich edge information in X-ray images to address the issue of insufficient texture capture. Finally, we employ an Adaptive Feature Interaction (AFI) module to obtain the optimal feature representation for final detection. Comprehensive experiments on the CLCXray and PIDray datasets demonstrate that HENet surpasses current state-of-the-art methods regarding detection accuracy.