This paper generates causal networks from raw text using Large Language Models (LLMs) and compares the accuracy and quality of generated networks with the ones produced by a recently suggested rule-based framework. The LLMs examined in this paper include ChatGPT, Mistral Chat, and Gemini. The evaluation was conducted on raw text drawn from three diverse domains: political, food insecurity, and medical. During the experiments, each LLM was given raw text belonging to these domains, and the generated causal networks were produced in the form of causal Subject-Verb-Object (SVO) triples. The quality of the produced network was assessed by a group of three human annotators. Comparative analysis reveals that, although LLMs show promise in generating causal networks, the rule-based system consistently demonstrates higher reliability in extracting precise causal relationships from raw text. The study highlights the importance of integrating traditional rule-based approaches with modern LLMs to enhance the correctness and comprehensiveness of causal network extraction, particularly in complex domains.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

On Comparing LLM-Generated Causal Networks with a Rule-Based Approach

  • Solat J. Sheikh,
  • Sajjad Haider

摘要

This paper generates causal networks from raw text using Large Language Models (LLMs) and compares the accuracy and quality of generated networks with the ones produced by a recently suggested rule-based framework. The LLMs examined in this paper include ChatGPT, Mistral Chat, and Gemini. The evaluation was conducted on raw text drawn from three diverse domains: political, food insecurity, and medical. During the experiments, each LLM was given raw text belonging to these domains, and the generated causal networks were produced in the form of causal Subject-Verb-Object (SVO) triples. The quality of the produced network was assessed by a group of three human annotators. Comparative analysis reveals that, although LLMs show promise in generating causal networks, the rule-based system consistently demonstrates higher reliability in extracting precise causal relationships from raw text. The study highlights the importance of integrating traditional rule-based approaches with modern LLMs to enhance the correctness and comprehensiveness of causal network extraction, particularly in complex domains.