Scholarlink

<p>Can GenAI evaluate scientific evidence? Large language models are becoming increasingly embedded across the research process, yet we lack direct tests of their ability to judge how empirical findings inform beliefs in a theory. I introduce a new benchmarking centered on Bayesian likelihood estimation, drawing from Bayesian process tracing, a case-based method in which researchers assign explicit values to how likely we would expect to observe a specific empirical finding given a theoretical hypothesis. Using 289 evidence–hypothesis pairs, I elicit likelihoods from Open AI’s GPT models. Com-pared to expert estimates, LLMs show substantial average distance, 30–40%, which is mostly upward bias, 20–30%. This divergence results in substantial difference in posterior ranking of competing hypotheses. Moving beyond the limited literature on process tracing, I extract 1390 evidence–hypothesis pairs from a diverse sample of 200 political science papers. In this broader corpus, models produce coherent and run-stable likelihood estimates, though likelihood continue to cluster at high levels. GPT-5 displays baseline robustness, correctly discarding irrelevant findings, showing minimal contamination from background knowledge of papers, and limited leakage from prior beliefs. Overall, while more capable models can produce likelihood judgments that are inter-nally coherent relative to Bayesian benchmarks, current state-of-the-art LLMs seem to consistently over-weight evidence.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Is Research Safe in the AI Revolution? LLMs Fall Short of Expert Benchmarks in Scientific Evidence Evaluation

  • 期刊论文

  • 发表日期：

    2026年2月28日

查看全文
  • Simone Paci

摘要

Can GenAI evaluate scientific evidence? Large language models are becoming increasingly embedded across the research process, yet we lack direct tests of their ability to judge how empirical findings inform beliefs in a theory. I introduce a new benchmarking centered on Bayesian likelihood estimation, drawing from Bayesian process tracing, a case-based method in which researchers assign explicit values to how likely we would expect to observe a specific empirical finding given a theoretical hypothesis. Using 289 evidence–hypothesis pairs, I elicit likelihoods from Open AI’s GPT models. Com-pared to expert estimates, LLMs show substantial average distance, 30–40%, which is mostly upward bias, 20–30%. This divergence results in substantial difference in posterior ranking of competing hypotheses. Moving beyond the limited literature on process tracing, I extract 1390 evidence–hypothesis pairs from a diverse sample of 200 political science papers. In this broader corpus, models produce coherent and run-stable likelihood estimates, though likelihood continue to cluster at high levels. GPT-5 displays baseline robustness, correctly discarding irrelevant findings, showing minimal contamination from background knowledge of papers, and limited leakage from prior beliefs. Overall, while more capable models can produce likelihood judgments that are inter-nally coherent relative to Bayesian benchmarks, current state-of-the-art LLMs seem to consistently over-weight evidence.