Pushing the Boundaries of Legal Information Processing with Integration of Large Language Models

The legal domain presents unique challenges in information processing, given the complexity and specificity of legal texts. Addressing these challenges, this work leverages breakthroughs in Large Language Models (LLMs) to push the boundaries in legal information extraction and entailment. Our approaches involve the integration of LLMs in the COLIEE 2024 competition across four tasks: Legal Case Retrieval (Task 1), Legal Case Entailment (Task 2), Statute Law Retrieval (Task 3), and Legal Textual Entailment (Task 4). In Task 1, we employ a two-stage strategy that combines keyword-based retrieval using BM25 with a sophisticated MonoT5 reranker fine-tuned on legal datasets. For Task 2, we further adapt MonoT5, incorporating hard negative sampling. For Task 3, we introduce a novel strategy that utilizes LLMs to enhance the performance of high-recall predictions from smaller language models, an approach we also adapt for Task 2. To address Task 4, we employ an ensemble of LLMs’ predictions, adjudicated via majority voting and the Dawid-Skene label model. Our strategies take advantage of the strengths of each model, with prompting techniques and constraints applied to exploit ensemble advantages. Consequently, we have achieved the top-ranked performance in Task 3 and secured promising outcomes in Tasks 1, 2, and 4. This paper describes our methodologies, offering insights into how integrating LLMs into legal information systems can significantly enhance their efficacy in tackling complex legal documents.