Semi-automatic Generation of Math Word Problems for Finding Reasoning Errors in Large Language Models
摘要
Mathematical Word Problem (MWP) is one of the tasks in natural language reasoning. MWP presents mathematical exercises as narratives, requiring the translation of textual descriptions into mathematical expressions. Unlike standard mathematical problems, MWP solvers must interpret and extract relevant information from the narrative. This process enables the evaluation of solvers across multiple skills, including reading comprehension, information identification, text-to-mathematics translation, and mathematical problem-solving. While reasoning performance did not follow a scaling law, Chain-of-Thought (CoT) has emerged to enable step-by-step reasoning, improving the performance of all reasoning tasks, including MWPs. However, most existing MWP datasets primarily focus on linguistic features such as lexical diversity, grammatical accuracy, or rely on augmenting already validated datasets through various data augmentation methods. As a result, they remain limited in terms of mathematical complexity and the diversity of the expressions presented. Consequently, examining the full potential of LLMs in tackling a broader array of mathematical concepts remains limited, and coverage for discovering hidden linguistic and reasoning deficiencies of LLMs is also limited. In this paper, to bridge these gaps, we propose generating MWP datasets by LLMs, incorporating a wider range of mathematical expressions and topics. The generated MWPs are followed by a manual verification process conducted by the researcher to ensure accuracy and quality. This approach enabled to generation of TEFMP, which contains 978 problems spanning around 13 diverse mathematical topics. Then LLMs are questioned to solve the generated problems and their variations. Through this, we reveal the weaknesses of current LLMs in solving math problems.