Towards Trustworthy Legal AI: A Multi-Agent Approach to Integrating Legislative Knowledge
摘要
Large language models exhibit advanced cognitive capabilities, enabling the development of AI assistants across domains. However, they face challenges in specialized fields like law, including hallucinations, the need for precise legal interpretation, and handling similar yet distinct legal systems across countries. This paper is a contribution to the field of development of reliable, hallucination-free legal LLM-based assistants. It introduces an AI assistant architecture designed to process multiple legal acts and improve the quality of legal responses. Implemented for Polish law, it incorporates knowledge from 49 key Polish legal acts but can be easily adapted to other statute-based legal systems. The architecture uses a multi-agent approach with three main components: query routing to specialized agents for specific legal acts, regulation retrieval, and reasoning to generate insightful responses. It employs a variant of the retrieval-augmented generation (RAG) approach to support efficient retrieval. We evaluate the solution using the 2024 attorney and legal advisers’ apprenticeship entry exam, consisting of 150 single-choice questions. Evaluation criteria include accuracy in providing correct answers and citing relevant legal regulations. We assess the architecture, its ablated versions, and compare it with commercial large language models. Our findings demonstrate a practical method for enhancing AI legal assistants by integrating multiple legal acts, paving the way for broader applications in national legal systems. This research also lays the foundation for analyzing court case files and providing legally compliant advice.