The proliferation of misinformation in the digital age has made fact-checking organizations critical for verifying claims and combating false information. However, fact-checking articles themselves may contain cognitive biases that influence how information is presented and conclusions are framed. The presence of these biases can undermine the credibility of fact-checking process. Previous research has focused on creating misinformation datasets, automating fact-checking article generation, and conceptually mapping cognitive biases relevant to journalism. However, these studies have not empirically examined the cognitive biases present in professionally written fact-checking articles, nor investigated how different large language models (LLMs) agree or disagree in detecting these biases. This study addresses these gaps by analyzing 60 fact-checking articles from four major Indian organizations using three state-of-the-art LLMs: GPT-4o-mini, Perplexity Sonar-Pro, and Google Gemini 2.5 Flash. We employed quantitative metrics including detection frequency, average bias count per article, inter-model agreement analysis, and signature bias preferences, while mapping detected biases to Dual Process Theory to understand cognitive exploitation patterns. Our results reveal only 7% tri-model consensus with detection rates varying 5–6 times across models, indicating substantial disagreements across LLMs. Analysis using Dual Process Theory shows a 17:1 System 1 to System 2 bias ratio, with 94.4% of biases targeting emotional rather than analytical thinking. Confirmation bias dominated at 31.4%, followed by illusory truth effect (20%) and availability heuristic (17.4%), suggesting fact-checking content employs similar persuasive techniques as those expressed in the spread of misinformation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Detection of Biases in Fact-Checking Using LLMs

  • Advika Thakur,
  • Ayushi Dubey,
  • Rishabh Kaushal

摘要

The proliferation of misinformation in the digital age has made fact-checking organizations critical for verifying claims and combating false information. However, fact-checking articles themselves may contain cognitive biases that influence how information is presented and conclusions are framed. The presence of these biases can undermine the credibility of fact-checking process. Previous research has focused on creating misinformation datasets, automating fact-checking article generation, and conceptually mapping cognitive biases relevant to journalism. However, these studies have not empirically examined the cognitive biases present in professionally written fact-checking articles, nor investigated how different large language models (LLMs) agree or disagree in detecting these biases. This study addresses these gaps by analyzing 60 fact-checking articles from four major Indian organizations using three state-of-the-art LLMs: GPT-4o-mini, Perplexity Sonar-Pro, and Google Gemini 2.5 Flash. We employed quantitative metrics including detection frequency, average bias count per article, inter-model agreement analysis, and signature bias preferences, while mapping detected biases to Dual Process Theory to understand cognitive exploitation patterns. Our results reveal only 7% tri-model consensus with detection rates varying 5–6 times across models, indicating substantial disagreements across LLMs. Analysis using Dual Process Theory shows a 17:1 System 1 to System 2 bias ratio, with 94.4% of biases targeting emotional rather than analytical thinking. Confirmation bias dominated at 31.4%, followed by illusory truth effect (20%) and availability heuristic (17.4%), suggesting fact-checking content employs similar persuasive techniques as those expressed in the spread of misinformation.