Autonomous driving emerges as a promising solution to address societal challenges. Driverless vehicles offer accessible transportation options for individuals of all ages and health conditions. Leveraging speed and trajectory control algorithms, these vehicles can execute optimised manoeuvres and seamlessly synchronise with other autonomous counterparts, thereby ensuring safe travel experiences and reduced energy consumption/pollution. An accurate interpretation of sensor data is a key element in the advancement of autonomous driving technology. Our current study explores object detection algorithms using convolutional neural networks. The primary focus lies in developing detectors tailored for multi-object tracking in video sequences. We propose integrating object tracking techniques into the detector’s training to evaluate if detections can be properly matched along tracks. An additional loss term compares current frame detections with those extracted from preceding frames, leveraging Kalman filters. This facilitates an in-depth analysis during training, exposing less accurate detections or false positives/negatives across successive frames. Consequently, the approach can ensure increased detection accuracy and robustness against transient disruptions such as shifts in illumination or partial obstruction of objects.