Can LLMs Guess Like Humans? Evaluating Inferential Reasoning Under Ambiguity
摘要
Recent advancements in Large Language Models (LLMs) have showcased impressive performance across factual and commonsense reasoning benchmarks. However, their capacity to perform inference under ambiguity and minimal context remains underexplored. We introduce a novel dataset of 50 general-purpose questions designed to evaluate single-hop inferential reasoning in settings where surface retrieval fails and contextual deduction is required. In a zero-shot, single-turn evaluation, ChatGPT-4 achieved only 20.8% accuracy substantially below human performance. While the model’s responses were syntactically fluent, they frequently lacked introspective depth and situational awareness. In contrast, human answers reflected emotional grounding, personal insight, and commitment to a singular, contextually plausible interpretation. Our findings expose critical limitations in current LLMs’ ability to simulate human-like reasoning and underscore the need for architectures that incorporate reflective and uncertainty-aware mechanisms.