Road traffic accidents claim millions of lives annually, majorly involving vulnerable road users such as pedestrians and cyclists. Proactive traffic scene risk assessment with priority to vulnerable traffic actors can drastically reduce the chances of collision and save lives. To improve the safety of advanced driver-assistance systems (ADAS), we propose Object Risk Estimation (ORE), a novel framework that combines motion, lane-position, and class labels to enable context-aware and ethically prioritized risk assessment. It uses a multitask framework that integrates object detection, Time-to-Contact (TTC) estimation, and semantic segmentation in a single deep neural network. Further, it uses post-processing to combine these outputs to estimate risk per object and overall scene risk. Leveraging a shared backbone, ORE achieves a \(27\%\) reduction in inference time while delivering superior performance, with 1.5 higher \(mAP\; \%\) and \(11\%\) lower oMiD loss on the Waymo dataset. Its modular design ensures transparency, with intermediate outputs directly explaining risk scores, eliminating the need for post-hoc analysis. By facilitating timely transfer of control and protecting vulnerable road users, ORE offers a practical, interpretable, and efficient solution for ADAS and autonomous driving safety, scalable to diverse traffic scenarios and higher autonomy levels.