A Comprehensive Ontology Knowledge Evaluation System for Large Language Models
摘要
Large Language Models (LLMs) have acquired vast amounts of knowledge through extensive pre-training on large corpora. However, the depth of their understanding of ontology knowledge remains unclear. Ontology is about what types of entities exist, how they are grouped into categories, and how they are related to one another. Ontology knowledge provides a standardized approach to knowledge representation that aligns with human cognition. This study aims to analyze the comprehension and mastery of ontological knowledge within LLMs. In this paper, we proposes a comprehensive benchmark that includes generative ontology knowledge data, focusing both on classes and properties. We develop an extensive evaluation framework and design various tasks aiming to evaluate the memorization and utilization of ontological knowledge. The experimental results indicate that, while LLMs can memorize ontological knowledge, they struggle with truly comprehending and utilizing it effectively.