PureBiasoMeter : Decoupling Popularity Bias from User Fairness in LLM-Based Recommender Systems
摘要
Large Language Models (LLMs) have transformed Recommendation Systems (RecSys) by enabling more user-aware interactions. However, this shift raises new challenges in evaluating fairness, particularly due to confounding systemic biases such as popularity bias. Conventional fairness assessments often confuse disparities in user treatment with systemic biases toward popular items, resulting in misleading conclusions. In this paper, we introduce PureBiasoMeter, a diagnostic framework that decouples popularity bias from user-level fairness in LLM-based RecSys. Our approach is based on the hypothesis that accurate fairness evaluation requires first mitigating popularity bias and then remeasuring fairness metrics. We generate a comprehensive set of 72 prompts using different user profiling strategies, demographic variants, and bias mitigation instructions. By applying these prompts in a black-box LLM setting, we evaluate fairness sensitivity, recommendation quality, and bias levels across multiple dimensions. Our results demonstrate that removing popularity bias substantially alters fairness measurements and reveals underlying disparities that were previously unknown. PureBiasoMeter thus provides a more reliable basis for fairness analysis and contributes a practical tool for disentangling intertwined sources of bias in modern RecSys.