Scholarlink

<p>We propose the task of <i>legal question generation</i> (QG) as an application in Legal NLP. Specifically, the task is to generate a question, given a context and an optional keyword. We create the first dataset for the QG task in the legal domain, called <i>LegalQ</i>, consisting of 2023 &lt;context, question&gt; pairs spanning the legal systems of multiple countries, and multiple languages. We then use this dataset to benchmark several Large Language Models (LLMs) including Turbo-GPT-3.5, GPT-4, Llama2-70b, Llama2-13b, and Aalap-Mistral-7b (a legal domain-specific LLM). We also fine-tune several open-source LLMs such as T5, BART, Pegasus, and Flan-T5, which helps to improve results over zero-shot prompting of LLMs. We also use the idea of in-context learning (via few-shot examples) to generate questions of varying types and difficulty levels. Furthermore, we introduce a novel domain-specific prompting strategy based on chain-of-thoughts prompting for question generation. Further, we perform Bloom Taxonomy analysis of the questions generated by the LLMs, thereby showing that ‘understanding’ and ‘remembering’ are the two most dominant types of questions generated by the LLMs. Human evaluation of the generated questions shows promise in terms of generating grammatically correct, relevant, appropriate, complex, and novel questions. Finally, we analyze the incomplete or unanswerable generated questions to find possible reasons for these issues.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Investigating legal question generation using large language models

  • 期刊论文

  • 发表日期：

    2025年4月15日

查看全文
  • Aniket Deroy,
  • Kripabandhu Ghosh,
  • Saptarshi Ghosh

摘要

We propose the task of legal question generation (QG) as an application in Legal NLP. Specifically, the task is to generate a question, given a context and an optional keyword. We create the first dataset for the QG task in the legal domain, called LegalQ, consisting of 2023 <context, question> pairs spanning the legal systems of multiple countries, and multiple languages. We then use this dataset to benchmark several Large Language Models (LLMs) including Turbo-GPT-3.5, GPT-4, Llama2-70b, Llama2-13b, and Aalap-Mistral-7b (a legal domain-specific LLM). We also fine-tune several open-source LLMs such as T5, BART, Pegasus, and Flan-T5, which helps to improve results over zero-shot prompting of LLMs. We also use the idea of in-context learning (via few-shot examples) to generate questions of varying types and difficulty levels. Furthermore, we introduce a novel domain-specific prompting strategy based on chain-of-thoughts prompting for question generation. Further, we perform Bloom Taxonomy analysis of the questions generated by the LLMs, thereby showing that ‘understanding’ and ‘remembering’ are the two most dominant types of questions generated by the LLMs. Human evaluation of the generated questions shows promise in terms of generating grammatically correct, relevant, appropriate, complex, and novel questions. Finally, we analyze the incomplete or unanswerable generated questions to find possible reasons for these issues.