Evaluation of the Effect of Side Information on LLM Rankers for Recommender Systems

Recently, Large Language Models (LLMs) have begun to be used in a wide variety of domains, marking their entry into the domain of recommender systems. Several approaches have been proposed to integrate LLMs into the recommendation process. One particularly promising approach is a two-stage system where a traditional recommender is first used to generate a candidate set, followed by an LLM acting as a ranker to refine the order of the top-k suggestions. This paper investigates how the results of training such an LLM ranker can be influenced by incorporating different side information about the candidates. The implications of these modifications are explored through a case study, with the results pointing to a consistent improvement of the retriever’s performance due to the action of the LLM ranker.