Relation labeling in product knowledge graphs with large language models for e-commerce

Product Knowledge Graphs (PKGs) play a crucial role in enhancing e-commerce system performance by providing structured information about entities and their relationships, such as complementary or substitutable relations between products or product types, which can be utilized in recommender systems. However, relation labeling in PKGs remains a challenging task due to the dynamic nature of e-commerce domains and the associated cost of human labor. Recently, breakthroughs in Large Language Models (LLMs) have shown surprising results in numerous natural language processing tasks, especially in the in-context learning (ICL). In this paper, we conduct an empirical study of LLMs for relation labeling in e-commerce PKGs, investigating their powerful learning capabilities in natural language and effectiveness in predicting relations between product types with few-shot in-context learning. We evaluate the performance of various LLMs, including PaLM-2, GPT-3.5, and Llama-2, on benchmark datasets for e-commerce relation labeling tasks. We use different prompt engineering techniques to examine their impact on model performance. Our results show that LLMs can achieve competitive performance compared to human labelers using just 1–5 labeled examples per relation. We also illustrate the bias issues in LLMs towards minority ethnic groups. Additionally, we show that LLMs significantly outperform existing KG completion models or classification methods in relation labeling for e-commerce KGs and exhibit performance strong enough to replace human labeling. Beyond empirical investigations, we also carry out a theoretical analysis to explain the superior capability of LLMs in few-shot ICL by comparing it with kernel regression.