Can Large Language Models Play to Win? Game-Theoretic Benchmarks in Poker for Probabilistic Reasoning Evaluation
摘要
Recent advances in Large Language Models (LLMs) have raised new questions about their abilities in reasoning and decision-making, particularly under uncertainty. While traditional evaluation benchmarks primarily focus on text understanding or deterministic tasks, they fail to probe the complex probabilistic reasoning and strategic planning essential for real-world intelligence. To bridge this gap, we propose PokerBench, a game-theoretic evaluation framework based on Texas Hold’em poker, for systematically assessing LLMs’ capacity for probabilistic reasoning and decision making under incomplete information. Experiments with several state-of-the-art LLMs reveal significant limitations in their strategic behavior when compared with established algorithmic baselines, highlighting the challenges that current models face in game-theoretic and real-world decision scenarios. These findings emphasize the need for more comprehensive and fine-grained evaluation methods to drive progress toward robust and general AI reasoning.