Large Language Models for Tracking Reliability of Information Sources

Human decision makers track variability in the reliability of their information sources, such that human decision making can be modeled using a hierarchical gaussian filter [1]. This variability tracking can be utilized to infer the intentions of social information sources, akin to Theory of Mind (ToM), which describes the ability to ascribe mental states to others [2]. Meanwhile, Large Language Models (LLMs) have shown evidence of an emergent ToM as larger and larger models move toward producing human-like performance on false belief Theory of Mind tasks [3]. While LLMs have shown evidence of ToM, do they behave like humans in other inferential abilities? Specifically, can a sufficiently large language model track variability in information sources? Here, we conduct an experiment to address this question, finding modest success among LLMs in identifying simple patterns in longitudinal data from an information source and in providing responses consistent with detecting a change in that source’s reliability. When more complex patterns are presented, however, the LLMs tested failed and overall provided responses that were non-human-like in a number of ways.