As artificial intelligence (AI) systems become increasingly integrated into our daily lives, ensuring their ethical behavior is paramount. This paper introduces a novel approach to embedding moral cognition in AI based on contractualism, an ethical theory that deems actions appropriate if they can be justified based on the set of norms that govern interactions between pairs of agents. Our model, embodied in a robotic cleaning agent named Sweepy, makes decisions based on a combination of predefined norms and dynamic user feedback. This hybrid decision-making process allows Sweepy to align its behavior with individual user preferences, thus enhancing user satisfaction. Through extensive simulations, we demonstrate Sweepy’s ability to learn and adapt its actions across different scenarios, illustrating the model’s robustness and flexibility. This study highlights the advantages of the contractualist approach over utilitarian and deontological models, particularly in their support of user autonomy and fairness. Our findings underscore the need to develop AI systems capable of making ethically sound decisions in complex, real-world environments, paving the way for more trustworthy and user-centric AI applications.