Understanding the Limitations of Large Language Models in Credibility-Tracking Tasks
摘要
Previous research has modeled humans’ ability to track changes in the credibility of an information source 17 (Diaconescu et al., 2014), but this has yet to be replicated in Large Language Models (LLMs). Recent studies have shown that LLMs generally exhibit poor abilities to reason about longitudinal data (Chandrasekaran et al., 2024; Zaroukian, 2024), but the prompting method used in these studies may result in the LLM referring only to the most recent data provided. In this study, we evaluate an LLM’s longitudinal reasoning capabilities by expanding upon this pervious work to test the LLM’s ability to reason about two data points, both before and after a change occurs in the information source’s reliability. We find that the LLM performs consistently worse when asked to reason about data points occurring earlier within the pattern and reveal the limitations of previous studies.