Domain Adaptation of Large Language Model Tokenizers—A Comparison of Performances
摘要
The advancement of large-scale language models in recent years has significantly enhanced various natural language processing (NLP) domains. This research addresses the specific challenge of developing BERT-based models tailored for domain-specific language modeling. Tokenization efficiency within BERT-based models is directly related to model efficiency which in turn motivated the research we present in this paper. This research aims to enhance the development of a domain-adapted large language model for the case of Serbian language. Our focus was the influence of training dataset content on the quality of the domain-adapted large language model tokenizer developed for masked language modelling task. In this paper, we will present a comparison of the tokenization performance for two tokenizers developed using different mixtures of data within two versions of a large language model named SrBERTa.