Scholarlink

The development of generative large language models (LLMs) opens up new possibilities in professional domains, including law. Tax law, in particular, poses unique challenges due to its dynamism, complexity, and susceptibility to errors (“hallucinations”). There is a need for a systematic evaluation of LLM capabilities in this context, especially for Polish tax law, and validation of tools supporting result verification. The aim of this study is a multi-faceted evaluation of the latest LLM in the analysis of Polish private tax ruling and assessment of the potential of the “LLM as a Judge” concept as a quality control tool. A two-stage experimental study was conducted. In Stage 1, four experts evaluated the quality of analyses for 100 tax interpretations performed by four LLM (Gemini 2.5 Pro/Flash, GPT-4o, DeepSeek R1) using a proprietary Legal Quality Index (LQI). In Stage 2, the model quality rankings established by experts were compared with the rankings created by the “LLM as a Judge” (based on the OpenAI 03 model), measuring concordance using Spearman’s rank correlation coefficient, Rank-Biased Overlap (RBO), and Cohen’s κ coefficient. The impact of the “critique + rerank” procedure on the quality of analyses was also examined. Stage 1 results showed varying model performance, with Gemini achieving the highest LQI. Stage 2 results showed very high concordance between expert evaluations and the “LLM as a Judge” (Spearman 0.861, RBO > 0.9 for low p, κ = 0.88). This confirms the reliability of automated evaluation, especially in identifying models from the top ranks. The “critique + rerank” procedure significantly improved the quality of the analyses. The obtained results suggest that LLM-based tools can serve as reliable support for quality control of tax law analyses, leading to process optimization and potential cost savings in law firms. Further research should focus on agent-based approaches, utilizing domain-specific knowledge bases, and analyzing the impact of AI on task completion time.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-criteria Assessment of the Capabilities of Latest Large Language Models in Tax Law Analysis and Validation of the ‘LLM as a Judge’ Concept

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Tomasz Strąk,
  • Mateusz Piwowarski,
  • Michał Tuszyński,
  • Kesra Nermend

摘要

The development of generative large language models (LLMs) opens up new possibilities in professional domains, including law. Tax law, in particular, poses unique challenges due to its dynamism, complexity, and susceptibility to errors (“hallucinations”). There is a need for a systematic evaluation of LLM capabilities in this context, especially for Polish tax law, and validation of tools supporting result verification. The aim of this study is a multi-faceted evaluation of the latest LLM in the analysis of Polish private tax ruling and assessment of the potential of the “LLM as a Judge” concept as a quality control tool. A two-stage experimental study was conducted. In Stage 1, four experts evaluated the quality of analyses for 100 tax interpretations performed by four LLM (Gemini 2.5 Pro/Flash, GPT-4o, DeepSeek R1) using a proprietary Legal Quality Index (LQI). In Stage 2, the model quality rankings established by experts were compared with the rankings created by the “LLM as a Judge” (based on the OpenAI 03 model), measuring concordance using Spearman’s rank correlation coefficient, Rank-Biased Overlap (RBO), and Cohen’s κ coefficient. The impact of the “critique + rerank” procedure on the quality of analyses was also examined. Stage 1 results showed varying model performance, with Gemini achieving the highest LQI. Stage 2 results showed very high concordance between expert evaluations and the “LLM as a Judge” (Spearman 0.861, RBO > 0.9 for low p, κ = 0.88). This confirms the reliability of automated evaluation, especially in identifying models from the top ranks. The “critique + rerank” procedure significantly improved the quality of the analyses. The obtained results suggest that LLM-based tools can serve as reliable support for quality control of tax law analyses, leading to process optimization and potential cost savings in law firms. Further research should focus on agent-based approaches, utilizing domain-specific knowledge bases, and analyzing the impact of AI on task completion time.