Scholarlink

<p>We aimed at evaluating the consistency in causal reasoning capabilities of large language models (LLMs) in disease-specific contexts, across the three levels of the causal ladder—associational, interventional, and counterfactual. We aimed at obtaining realistic clinical scenarios involving drug interactions and side effects for Human Immunodeficiency Virus (HIV) antiretroviral therapy (ART). LLMs were prompted with context-rich clinical and causal vignettes to generate use cases, followed by context-simple queries and context-enhanced ones with the provision of extra documentation (either accessible or inaccessible), to test hallucinations. Multiple open- and closed-source LLMs, including GPT-4o and LLaMA-3, were evaluated for intra- and inter-model agreement (Fleiss’ kappa), predictive performance with respect to generated scenarios (e.g., sensitivity, specificity), reasoning quality, alignment with clinical pharmacoepidemiology experts (Likert rates, Kendall’s W), stratified by causal level. LLMs exhibited moderate to high intra-model agreement, decreasing from associational to counterfactual queries, whilst inter-model agreement was notably lower. References to inaccessible online content were produced. Experts revealed modest alignment with LLM outputs, diminishing confidence in the ability of LLM to generate reliable scenarios and ground truth. Associational reasoning was rated most favorably, whilst counterfactual scenarios demonstrated key weaknesses in clinical realism and depth. In conclusion, while generally concordant, LLMs are not fully competent in causal reasoning within HIV clinical pharmacoepidemiology.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Consistency in causal reasoning for large language models in scenarios of HIV antiretroviral treatment, drug interactions, and side effects

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月27日

查看全文
  • Mattia Prosperi,
  • Simone Rancati,
  • Yi Guo,
  • Jingchuan Guo,
  • Rotana M. Radwan,
  • Marco Salemi,
  • Jiang Bian,
  • Balu Bhasuran,
  • Eric F. Egelund,
  • Jennifer Janelle,
  • Preeti Manavalan,
  • Nicole Maranchick,
  • Simone Marini,
  • Zhe He

摘要

We aimed at evaluating the consistency in causal reasoning capabilities of large language models (LLMs) in disease-specific contexts, across the three levels of the causal ladder—associational, interventional, and counterfactual. We aimed at obtaining realistic clinical scenarios involving drug interactions and side effects for Human Immunodeficiency Virus (HIV) antiretroviral therapy (ART). LLMs were prompted with context-rich clinical and causal vignettes to generate use cases, followed by context-simple queries and context-enhanced ones with the provision of extra documentation (either accessible or inaccessible), to test hallucinations. Multiple open- and closed-source LLMs, including GPT-4o and LLaMA-3, were evaluated for intra- and inter-model agreement (Fleiss’ kappa), predictive performance with respect to generated scenarios (e.g., sensitivity, specificity), reasoning quality, alignment with clinical pharmacoepidemiology experts (Likert rates, Kendall’s W), stratified by causal level. LLMs exhibited moderate to high intra-model agreement, decreasing from associational to counterfactual queries, whilst inter-model agreement was notably lower. References to inaccessible online content were produced. Experts revealed modest alignment with LLM outputs, diminishing confidence in the ability of LLM to generate reliable scenarios and ground truth. Associational reasoning was rated most favorably, whilst counterfactual scenarios demonstrated key weaknesses in clinical realism and depth. In conclusion, while generally concordant, LLMs are not fully competent in causal reasoning within HIV clinical pharmacoepidemiology.