The Knowledge Graph (KG) is one of the knowledge representation techniques used to represent the knowledge present in the structured or unstructured text. The KG can be generated using various techniques and models, including rule-based, deep learning, and Large Language Model-based approaches. An assessment methodology is required to determine optimal KG generation methods. This paper proposes an evaluation algorithm for KG generation methods based on existing metrics Precision, Recall, and F1-Score, along with proposed metrics Average Similarity, Knowledge Score (K-Score), and Information Score (I-Score). The existing metrics Precision, Recall, and F1-Score metrics measure prediction accuracy concerning a particular task or dataset. Still, they do not consider the total amount of knowledge imparted uncertainty or information content of the knowledge graph. Therefore, the information uncertainty and the knowledge conveyed are assessed using the proposed metrics Average Similarity, K-Score, and I-Score. The proposed algorithm comprehensively evaluates the triplets generated from three KG generation methods. Our experimental findings demonstrate that the evaluation algorithm effectively and quantitatively differentiates between the various knowledge graph generation methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating Knowledge Graph Triplets: Methods and Metrics

  • Chaithra,
  • Sathyanarayana K. B.,
  • Biju R. Mohan,
  • Dinesh Naik

摘要

The Knowledge Graph (KG) is one of the knowledge representation techniques used to represent the knowledge present in the structured or unstructured text. The KG can be generated using various techniques and models, including rule-based, deep learning, and Large Language Model-based approaches. An assessment methodology is required to determine optimal KG generation methods. This paper proposes an evaluation algorithm for KG generation methods based on existing metrics Precision, Recall, and F1-Score, along with proposed metrics Average Similarity, Knowledge Score (K-Score), and Information Score (I-Score). The existing metrics Precision, Recall, and F1-Score metrics measure prediction accuracy concerning a particular task or dataset. Still, they do not consider the total amount of knowledge imparted uncertainty or information content of the knowledge graph. Therefore, the information uncertainty and the knowledge conveyed are assessed using the proposed metrics Average Similarity, K-Score, and I-Score. The proposed algorithm comprehensively evaluates the triplets generated from three KG generation methods. Our experimental findings demonstrate that the evaluation algorithm effectively and quantitatively differentiates between the various knowledge graph generation methods.