Scholarlink

Large Language Models (LLMs) are expected to exert unprecedented impact on the public sector. Specifically in the field of legal interpretation, the ability of LLMs to utilize the wealth of public sector information, such as legal texts and regulations, enables more efficient and accurate legal analysis and improves decision-making. However, when involving LLMs in such critical domains, it is important to ensure that they are trustworthy and, hence, they produce accurate responses. This study aims to explore and evaluate the trustworthiness of LLMs in interpreting law. Towards this direction, an exploratory case study is presented that engages nine proprietary and open LLMs from four families, namely Claude, GPT, Mistral, and Llama, in answering questions related to the European Union’s VAT directive. The set of questions has been selected by a legal professional. They are all of a legal nature and varying complexity and are provided as prompts to the LLMs. The legal precision of their responses is then evaluated. The results show significant insights, contributing to the development of more trustworthy and responsible AI systems and ensuring their safe and effective use in critical areas such as law and public policy.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating Open and Proprietary Large Language Models in Law Interpretation: The Case of the EU VAT Directive

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Areti Karamanou,
  • Evangelos Kalampokis,
  • Fotios Fitsilis,
  • Georgios Theodorakopoulos,
  • Konstantinos Tarabanis

摘要

Large Language Models (LLMs) are expected to exert unprecedented impact on the public sector. Specifically in the field of legal interpretation, the ability of LLMs to utilize the wealth of public sector information, such as legal texts and regulations, enables more efficient and accurate legal analysis and improves decision-making. However, when involving LLMs in such critical domains, it is important to ensure that they are trustworthy and, hence, they produce accurate responses. This study aims to explore and evaluate the trustworthiness of LLMs in interpreting law. Towards this direction, an exploratory case study is presented that engages nine proprietary and open LLMs from four families, namely Claude, GPT, Mistral, and Llama, in answering questions related to the European Union’s VAT directive. The set of questions has been selected by a legal professional. They are all of a legal nature and varying complexity and are provided as prompts to the LLMs. The legal precision of their responses is then evaluated. The results show significant insights, contributing to the development of more trustworthy and responsible AI systems and ensuring their safe and effective use in critical areas such as law and public policy.