Given the substantial potential of Large Language Models (LLMs) and the critical role that law plays in everyday life, a robust and comprehensive evaluation framework for legal performance is essential. To address the limitations of existing benchmarks in the judicial domain, we developed Legal-Eval, a comprehensive benchmark specifically designed for the Chinese judicial context, which incorporates 11 tasks from the CAIL competition. To facilitate instruction-based conversions, we established a unified data format and created 11 distinct instruction-based templates aligned with the input-output format typical of LLMs. We conducted an extensive evaluation of the LLM’s performance, comparing it to the Third-place results from the CAIL competition for each task. The results reveal that certain models exhibit performance in memory and comprehension tasks that approach or even exceed the Third-place score. However, in tasks requiring advanced discrimination abilities, such as Case Matching and Argument Understanding, even the highest-performing models demonstrate significant gaps. To address these challenges, we further investigated these tasks and developed the GLM4-Legal model through instruction tuning. This model outperformed the Third-place score in CAIL on five tasks. These findings contribute valuable insights for the ongoing development of LLMs in the legal domain.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Exploring the Capabilities of Chinese LLMs in the Legal Field

  • Zhengying Wang,
  • Jiong Yu,
  • Zheng Chu

摘要

Given the substantial potential of Large Language Models (LLMs) and the critical role that law plays in everyday life, a robust and comprehensive evaluation framework for legal performance is essential. To address the limitations of existing benchmarks in the judicial domain, we developed Legal-Eval, a comprehensive benchmark specifically designed for the Chinese judicial context, which incorporates 11 tasks from the CAIL competition. To facilitate instruction-based conversions, we established a unified data format and created 11 distinct instruction-based templates aligned with the input-output format typical of LLMs. We conducted an extensive evaluation of the LLM’s performance, comparing it to the Third-place results from the CAIL competition for each task. The results reveal that certain models exhibit performance in memory and comprehension tasks that approach or even exceed the Third-place score. However, in tasks requiring advanced discrimination abilities, such as Case Matching and Argument Understanding, even the highest-performing models demonstrate significant gaps. To address these challenges, we further investigated these tasks and developed the GLM4-Legal model through instruction tuning. This model outperformed the Third-place score in CAIL on five tasks. These findings contribute valuable insights for the ongoing development of LLMs in the legal domain.