LLM Tuning and Interpretable CoT: KIS Team in COLIEE 2024

Focusing on the recently advanced Large Language Models (LLMs), we studied two key areas: LLM tuning and Chain-of-Thought (CoT) interpretability, which are crucial in legal tasks. Regarding LLM tuning, we conducted experiments comparing multiple models, a variety of techniques including prompt engineering, and the presence or absence of fine-tune, thereby identifying the most effective settings. Additionally, we proposed new methods to overcome the shortcomings of models during fine-tune, leading to improved accuracy. In terms of CoT interpretability, we introduced a format that facilitates guiding the CoT process, enabling the identification of which parts of the reasoning process significantly influence the inference results. Furthermore, we demonstrated that fine-tune enables any model to produce outputs in this format, and this approach can clearly define the differences in reasoning capabilities among various models. Using implication reasoning tasks in the civil law section of the judicial examination as a subject, we achieved enhanced task performance with large-scale language models.