Tabular reasoning presents a significant challenge in understanding natural language queries in the context of provided tables, mainly because of the complex logical operations involved. Pre-trained language models have demonstrated their capabilities in various tasks. However, performing pre-training specifically for tabular reasoning is difficult due to the diverse range of reasoning abilities required beyond contextual understanding. In this work, we propose Tabular Reasoning with Two-stage Knowledge Injection (TsKI). TsKI consists of two components: TsKI \(_{\textsc {Stage1}}\) and TsKI \(_{\textsc {Stage2}}\) . The primary objective of TsKI \(_{\textsc {Stage1}}\) is to incorporate symbolic knowledge into pre-trained language models by utilizing synthesized programs. It begins by generating high-quality programs using a specific program synthesis algorithm. Next, TsKI \(_{\textsc {Stage1}}\) conducts pre-training on the automatically generated corpus, enabling the model to learn how to query tables using the generated programs. On the other hand, TsKI \(_{\textsc {Stage2}}\) aims to inject step-wise knowledge into the model. It starts by decomposing natural language queries into multiple sub-queries using heuristic rules and a constituency parser. Then, it employs pre-trained language models themselves to query tables with the obtained sub-queries, obtaining intermediate results that facilitate step-wise tabular reasoning. Experimental results demonstrate the effectiveness of our proposed approach. TsKI achieves significant improvements on two well-known tabular reasoning datasets, namely TabFact and WikiTableQuestions, in both TsKI \(_{\textsc {Stage1}}\) and TsKI \(_{\textsc {Stage2}}\) . Furthermore, in-depth analysis validates the effectiveness of each component of our approach. The code is available at https://github.com/qshi95/TsKI.
