Adaptation of Large Language Models for the Public Sector: A Clustering Use Case

Recent research has highlighted the potential of domain adaptation in improving the performance of generic Large Language Models (LLMs) [1, 5, 19, 20]. In this study, we aim to explore the benefits of LLMs and their adaptation on language and domain-specific data for the public sector. Our focus is on investigating the impact of domain adaptation for a specific use case in the European public service: the clustering of pledges on the Transition Pathway for Tourism (see here ). First, a limited corpus of official documents and legislation on the Transition Pathway for Tourism was collected. Then, relying on existing approaches for domain adaptation of large language models, this corpus was used to adapt two pre-trained language models (BERT and RoBERTa) on the domain of interest. Finally, an innovative approach based on Azure OpenAI GPT4 as a human emulator was used to evaluate the impact of domain adaptation. The results of our study revealed a nuanced impact of domain adaptation. While the domain-adapted LLMs did not generate quantitatively more coherent clusters compared to their pre-trained counterparts, they exhibited a positive impact on the accuracy of the model (at a level of 5%) when considering the qualitative aspect of the clusters’ content. This suggests that domain adaptation can enhance the interpretability and usability of the clusters, even when working with a small dataset. However, it is worth noting that, in terms of interpretability and usability, a simpler model like Word2Vec outperformed the LLMs even after domain adaptation.