Finding Our Moral Values: Guidelines for Value System Aggregation

Ensuring AI aligns with our moral values is an important problem. To achieve AI alignment though, we first need to know what values we want AI to align with. However, obtaining a model of society’s values is far from a simple problem. This problem, called value inference, typically needs to perform value system aggregation which consists in combining moral value models of several individuals to obtain one representing everybody. So far, only one such method has been proposed. This paper discusses why research in value system aggregation is necessary and outlines avenues to implement it depending on the value alignment problem at hand. In particular, we discuss how value system aggregation can be useful for policymaking applications.