Distribution-Aware Diversification for Personalized Re-ranking in Recommendation

Improving the diversity of recommendation systems plays an important role in enhancing user experience and mining user potential interests. The re-ranking stage, as the final stage of the recommendation system, has a direct impact on the recommendation results. Many works dedicate to improving the diversity of recommendation systems in the re-ranking stage, but most of them optimize diversity based on traditional pairwise distance between elements in the list while overlooking the listwise category distribution of the list. Besides, most of re-ranking methods do not considered users’ personal preferences towards the diversity, and diversify the list equally for all users. In this work, we propose Distribution-aware Diversification for Personalized Re-ranking, which integrates listwise category distribution and user personalization into the optimization objective of diversity, increasing the number of categories in the list while making the distribution of categories more balanced and personalized than traditional re-ranking methods. We conduct experiments on two public datasets and one private dataset, the results demonstrate that our proposed method effectively improves the recommendation diversity in the re-ranking stage while maintaining the accuracy.