Autonomous vehicles rely on deep learning-based object detection algorithms to sense the environment and make critical decisions, thus being highly susceptible to adversarial attacks. These attacks can subtly alter visual inputs that lead to severe misclassifications while being resilient in practical environments, thereby threatening the reliability of autonomous perception systems. In this work, we introduce a training-free adversarial defense approach using semantic similarity reasoning that can detect and rectify adversarial-tainted object detection results without the need to update the model. The premise is to check if the detected object semantically aligns with the predicted category by comparing the representation with benchmark exemplars, thereby identifying the semantic discrepancies created by adversarial attacks. By integrating semantic information with spatial information and temporal cue consistency among video frames, the proposed approach effectively distinguishes adversarial patterns from benign ones. Once an adversarial pattern is recognized, the approach derives the most likely true category through similarity search, enabling immediate correction of the misclassification at the inference stage. Experiments on adversarial patch attacks on real-time object detection prove the effectiveness of semantic consistency-based defense in enhancing reliability in safety-critical autonomous systems.