Object detection and tracking are crucial tasks in the field of computer vision, especially in real-time scenarios where multiple objects need to be detected, localized, and tracked with temporal accuracy. Existing models like YOLO and Faster R-CNN, though effective, often struggle with challenges related to computational efficiency and real-time performance. In this study, we introduce an advanced approach that combines YOLOv5 for fast object detection, DeepSORT for precise object tracking, and transformers for improved contextual comprehension. By utilizing YOLOv5’s rapid detection capabilities, DeepSORT’s deep appearance features, and the attention mechanisms of transformers, our system enhances both detection and tracking accuracy, especially in dense and complex environments. Furthermore, EfficientDet is incorporated to boost detection precision, while BERT is utilized for robust query understanding and object matching. This combination ensures high computational efficiency alongside improved accuracy, making it ideal for real-time applications such as autonomous driving and surveillance. Our proposed solution effectively addresses the limitations of conventional models, providing a robust and accurate system for real-time object detection and tracking. Our system attained an average detection accuracy of 94.48%, with a tracking precision of 90.7%, evaluated on custom surveillance video datasets. The system also has a real-time processing speed of 42 FPS (frames per second), which makes it ideal for applications such as autonomous driving and surveillance.