User Fairness in Recommender Systems Using Beyond-Accuracy Basket Quality Metrics
摘要
Recommender systems (RecSys) have conventionally assessed user fairness through accuracy-based metrics, such as Root Mean Squared Error and Hit Rate. These metrics are limited in their ability to evaluate fairness of user experiences, as they do not consider beyond-accuracy perspectives of recommendation basket quality. Key elements of beyond-accuracy perspectives - including novelty, relevancy, unexpectedness, coverage, serendipity, ranking, diversity, and recency - collectively enhance existing frameworks. They do so by enabling evaluation of user satisfaction and user fairness across different demographic groups. Expanding upon established fairness metrics in the literature, this work introduces novel beyond-accuracy basket quality perspectives, including novelty-relevancy (NovRel) and recency, along with complementary metrics for computing relevancy and unexpectedness at individual and group levels. While leveraging existing accuracy-based fairness measures, our approach evaluates user fairness through the lens of beyond-accuracy basket quality measures and provides a comprehensive assessment of whether RecSys deliver comparable experiences to similar users and ensure equitable treatment across different sensitive groups. Methods to calculate these metrics are demonstrated using common algorithms, SVD++, Q-SVD++, and MLP-SVD++, which are trained on two widely used benchmark datasets. The trade-off between accuracy metrics and beyond-accuracy fairness metrics is presented. Beyond-accuracy metrics provide valuable insights into how these algorithms differ in user fairness due to their inherent design considerations.