This chapter explores the implementation of the preference learning requirement within the framework of Dynamic Normativity. Central to this requirement is the assumption that AI systems can align with human preferences when such preferences are embedded in their learning signals. Here, we investigate how normative information, expressed through human behavior and judgment, can shape an AI’s optimization process, guiding it toward outcomes aligned with human values. However, we also argue that this approach remains insufficient as a complete solution to the alignment problem while presenting common vulnerabilities inherent in current methods used for value learning.