Lessons Learned on Summarizing Legal Documents Combining Reinforcement Learning and ChatGPT

In legal institutions, the efficient access to information within vast amounts of documents is crucial. Existing search engines fail to provide an overview of these documents, leading to an arduous task of skimming through tens or even hundreds of pages. This paper presents a study aimed at addressing this issue by employing automatic text summarization. This task is complex due to the length of documents and the limited dataset available for training summarization models in Brazilian Portuguese. We experimented with different summarization approaches, gradually increasing the complexity to address the shortcomings of previous attempts. Our final solution is a hybrid model that employs a pipeline of a fine-tuned MemSum model using Reinforcement Learning to select the most relevant sentences and connecting its output to ChatGPT to generate a more comprehensible and fluent summary. Our results exhibited impressive performance, with Human Evaluation assigning an overall performance rating of ‘good’ or ‘very good’ to 98.7% of the categories using TAC methodology.