Generally, various handheld weapons, such as guns, swords, knives, etc., are used in criminal activities. Further, real-time detection of these weapons using intelligent video surveillance systems can act as a deterrent and legal evidence in smart city applications. Hence, this paper proposes a compact and efficient weapon detector based on the You Only Look Once (YOLOv5)-GhostNet model. A new weapon dataset, “Weapon7,” comprises seven weapon classes such as Axe, Bow and Arrow, Gun, Kinfe, Lathi, Pistol, and Sword, with proper annotation files have been developed. Experimental analysis shows that the proposed model performs better than the equivalent reported works in terms of online performance metrics such as precision, recall, mAP, FPS, and GFLOPS.