Driver distraction remains a significant contributor to road accidents; however, existing deep-learning detectors either sacrifice accuracy for speed on resource-constrained hardware or lose generality when confronted with unseen data. This work presents an end-to-end single-stage model that, in one forward pass, jointly identifies the driver, links the driver’s body parts, recognises nearby in-cabin objects, and determines whether the driver is distracted. By embedding spatial and semantic relationships directly into its output vector, the model avoids slow post-processing and significantly reduces false alarms caused by objects that merely appear near the driver. Evaluated on five public datasets and an additional real-world collection that were not used for training, the proposed detector boosts mean F1-score by 0.11 ( \(\approx \) 20% relative) over a lightweight baseline while maintaining 39 frames per second on an NVIDIA Jetson Xavier edge device—more than three times faster than a comparable two-stage pipeline. These results demonstrate a driver-distraction detector that simultaneously achieves cross-dataset robustness, real-time performance, and efficient deployment on low-power hardware.