Overview of Benchmark Datasets and Methods for the Legal Information Extraction/Entailment Competition (COLIEE) 2024

We summarize the 11th Competition on Legal Information Extraction and Entailment (COLIEE 2024). In this eleventh edition, the competition included four tasks on case law and statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and a selected unseen case (Task 2). The statute law component includes an information retrieval task (Task 3), and an entailment/question-answering task based on retrieved civil code statutes (Task 4). Participation was open to any group based on any approach. Ten different teams participated in the case law competition tasks, most of them in more than one task. We received results from 10 teams for Task 1 (26 runs) and 6 teams for Task 2 (18 runs). On the statute law task, there were 12 different teams participating, most in more than one task. 8 teams submitted a total of 20 runs for Task 3, and 8 teams submitted a total of 23 runs for Task 4. We describe the variety of approaches, our official evaluation, and analysis of our data and submission results.