Object detection technology is pivotal for autonomous vehicles to perceive their surroundings, focusing on static road traffic objects like traffic lights, signs, and pavement markings. Training separate networks for each detection task is resource-inefficient under constrained conditions. In order to address this issue, our research introduces the UniDet model, a unified framework for multi-task object detection. It efficiently handles multiple tasks with resource usage and inference speeds close to those of single-task models, outperforming the baseline by 7.1% in Average Precision (AP) on traffic datasets. We observed that multi-task detection often faces data and feature conflicts. To address these challenges, we developed a task decoupling strategy for collaborative cross-task learning. This mitigates learning conflicts due to inadequate data annotations and resolves feature conflicts between classification and localization, leading to a 2.2% performance boost. Additionally, the prevalent issue of detecting small objects was addressed by introducing multi-scale feature fusion and loss functions. A global context-aware aggregation module enhances feature extraction for small objects, while an instance scale-aware weight redistribution strategy tackles the learning imbalance, cumulatively increasing the model’s AP by 1.4% and specifically by 3.1% for small objects. This streamlined approach ensures methodological rigor and emphasizes the model’s innovations and contributions to the field of autonomous vehicle perception.