Exploring Cognitive Biases in LLM Predictions: Probability Matching in GPT-4o Mini
摘要
A variety of biases and heuristics shape human decision-making. When training artificial reasoning systems on corpora that include their use, the decisions made by these systems may then reflect these biases and heuristics. The work presented here explores the extent to which the phenomenon of probability matching is present in decisions made by GPT-4o mini. Results show no clear evidence of probability matching nor an optimal maximizing strategy. Instead, GPT-4o mini’s behavior is consistent with previous results and shows an inability to perceive and reason over the base frequencies accurately. Still, there appears to be a compounding effect of domain-related biases about probabilities and whether frequencies are presented as summarized counts or as individual event outcomes. This behavior is plausibly due to patterns in the training data in the former case and limitations of counting and reasoning via statistical association in the latter.