This study introduces a novel vehicle detection and classification method using YOLOv11, an advanced deep learning architecture integrated with powerful object annotation algorithms, namely BoT-SORT and ByteTrack. Conventional vehicle detection systems often struggle in dynamic environments characterized by fluctuating lighting, weather variations, and intricate vehicle movements. The proposed approach overcomes these issues through YOLOv11’s improved feature extraction and object detection capabilities, enabling consistent performance across real-world traffic conditions. The inclusion of BoT-SORT and ByteTrack facilitates precise and efficient real-time vehicle annotation, particularly in congested traffic scenes. This integrated system can continuously detect, classify, and annotate vehicles, offering a holistic solution tailored to intelligent transportation infrastructure. Experimental evaluations confirm its effectiveness, showing marked gains in detection accuracy, classification fidelity, and annotation consistency compared to conventional approaches. Our system achieved an accuracy of 0.99 using a dataset comprising 370 images from a top-down angle, which is already widely used in traffic surveillance systems. Furthermore, the efficacy in F1 score (1.00) and mean Average Precision (mAP) of 0.96 indicates the extensive ability of the proposed system for multi-label vehicle classification under challenging conditions. The findings emphasize the strength of merging state-of-the-art deep learning with sophisticated annotation technologies for practical traffic surveillance systems.