There has been tremendous growth in Autonomous Vehicles (AVs) recently, yet the moral decision-making capabilities remain a crucial challenge that needs to be addressed. Addressing this issue is pivotal for gaining societal trust towards AVs, as the decision-making of the AVs will impact human life in a significant manner. Incorporating human preferences in the decision-making of AVs ensures ethical decisions along with societal acceptance of decisions made under moral uncertainty. In this paper, we propose integrating human preferences into Reinforcement Learning (RL) to guide AVs to make decisions that resonates with human-values. We use the Bradley-Terry (BT) model to incorporate human preferences and perform pairwise comparisons on the moral machine framework of AVs. This approach of considering human preference adds a layer of explainability to the decisions and enhances the significance of the results for real-world applicability. The results show the decision-making capability of RL agents could be improved by embedding human preferences and the decisions made by AVs align closely with those of humans.