Object detection plays a critical role in numerous applications, including security surveillance and autonomous driving, where system reliability is crucial. However, modern object detectors, which are based on deep learning, are vulnerable to adversarial attacks that can severely degrade their performance. This paper proposes three key strategies to enhance the robustness of object detection models: (1) a confidence loss mechanism in multi-task learning settings to strengthen overall model robustness, (2) an optimized adversarial training method using fast adversarial training technique to enhance training efficiency, and (3) a class-wise adversarial training approach to ensure robustness is balanced across object classes, preventing weaker classes from being disproportionately affected. Extensive experiments and multiple ablation studies on the PASCAL-VOC and MS-COCO datasets confirm that these strategies significantly improve model robustness.