With the advancement of the rule of law and the growing public awareness of legal issues, the demand for high-quality legal services has been steadily increasing. Legal question-answering (QA) systems, which involve complex legal provisions and terminology, impose higher requirements on the accuracy and rigor of their responses. However, current large language models (LLMs) still face significant challenges in achieving automated and accurate evaluation of answer quality in legal QA scenarios. Incomplete or semantically ambiguous responses often lead to high costs for manual scoring. To address this challenge, this paper proposes an automated quality evaluation framework-KCBGE-CASL, which integrates a ContraNorm module and a cyclic asymmetric focal loss (ASL), and is built upon a BGE model with K-center clustering. The proposed method employs a dual-tower architecture to optimize semantic alignment between queries and answers, and introduces a concatenation difference module and novel loss function to enhance learning from imbalanced samples. Meanwhile, the K-Center-Greedy algorithm is utilized to cluster QA data, mitigating the problem of uneven topic distribution; the integration of the ContraNorm module further improves the discriminability of word embeddings. In addition, the framework incorporates a Direct Preference Optimization (DPO) mechanism, which directly models human preferences and effectively enhances the ranking ability for high-quality legal QA results. Experimental results demonstrate that the proposed method significantly outperforms mainstream approaches in the accuracy of legal QA quality evaluation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

KCBGE-CASL: An Automated Legal Question Answering Quality Evaluation Framework Integrating ContraNorm and Direct Preference Optimization

  • Jiawei Shi,
  • Dawei Liu,
  • Pengyun Bai,
  • Yanzhe Zhao,
  • Xiaolong Ding,
  • Yudi Wei

摘要

With the advancement of the rule of law and the growing public awareness of legal issues, the demand for high-quality legal services has been steadily increasing. Legal question-answering (QA) systems, which involve complex legal provisions and terminology, impose higher requirements on the accuracy and rigor of their responses. However, current large language models (LLMs) still face significant challenges in achieving automated and accurate evaluation of answer quality in legal QA scenarios. Incomplete or semantically ambiguous responses often lead to high costs for manual scoring. To address this challenge, this paper proposes an automated quality evaluation framework-KCBGE-CASL, which integrates a ContraNorm module and a cyclic asymmetric focal loss (ASL), and is built upon a BGE model with K-center clustering. The proposed method employs a dual-tower architecture to optimize semantic alignment between queries and answers, and introduces a concatenation difference module and novel loss function to enhance learning from imbalanced samples. Meanwhile, the K-Center-Greedy algorithm is utilized to cluster QA data, mitigating the problem of uneven topic distribution; the integration of the ContraNorm module further improves the discriminability of word embeddings. In addition, the framework incorporates a Direct Preference Optimization (DPO) mechanism, which directly models human preferences and effectively enhances the ranking ability for high-quality legal QA results. Experimental results demonstrate that the proposed method significantly outperforms mainstream approaches in the accuracy of legal QA quality evaluation.