Answering Spatial Commonsense Questions Based on Chain-of-Thought Reasoning with Adaptive Complexity

This paper focuses on answering the spatial questions in the task of machine reading comprehension (MRC-QA), which involves complex commonsense reasoning. Current mainstream methods are based on the large language model (LLMs) which uses the chain-of-thought (CoT) to support reasoning. However, these methods neglect to consider the differences in reasoning complexity of the questions when designing the CoT prompts, resulting in poor performance. Spatial questions involve complex positional relations and vast implicit commonsense knowledge. A simple single-hop prompt cannot extract enough implicit spatial knowledge from the LLMs to derive correct answers. Respectively, overly complex prompts for simple questions can mislead LLMs into over-reasoning, leading to wrong results. To address this problem, we propose a new framework with complexity-aware adaptive CoT, called CACoT. It can adjust the number of multi-hop steps according to the reasoning complexity of the given question. In detail, we first measure the reasoning complexity of the question. We then construct the demonstrations that fit the complexity. We retrieve demonstrations that match the test question complexity to build an adaptive CoT prompt. We further design a diversity thinking strategy to avoid insufficient reasoning. Experimental results on typical spatial datasets show the effectiveness of our method.