TaxBen: Benchmarking the Chinese Tax Knowledge of Large Language Models
摘要
Large Language Models (LLMs) excel in various areas but struggle in the highly specialized, knowledge-intensive, and legally regulated Chinese tax domain, where performance gaps remain. To address this, we introduce TaxBen, the first benchmark designed for evaluating Chinese tax LLMs. It assesses LLMs’ tax capabilities on three cognitive levels based on Bloom’s taxonomy: (1) Knowledge Memorization, (2) Knowledge Understanding, and (3) Knowledge Application. TaxBen includes 5 diverse specific tasks–single&multiple-choice, cloze test, numerical reasoning, summary generation, and Q&A–drawn from 9 datasets totaling 6K instances. We conducted an extensive evaluation of 18 LLMs on TaxBen, including 6 multilingual LLMs, 11 Chinese-focused LLMs, and 1 tax-related LLM. Results show that reveals significant performance differences: the closed-source ERNIE-3.5 excels, while open-source LLMs like Qwen2.5 and Yi underperform. YaYi2, fine-tuned with some tax data, shows limited improvement. TaxBen is a crucial resource for advancing LLM research and applications in the Chinese tax domain.