Scholarlink

Abstract <p>Large language models (LLMs) pretrained on English-centered corpora have biases and perform sub-optimally on other natural languages. Adaptation of LLMs vocabulary provides a resource-efficient way to improve the quality of a pretrained model. Previously proposed adaptation techniques focus on performance (accuracy) and size metrics (fertility), ignoring other aspects in comparison, such as inference latency, compute resources for adaptation, and catastrophic forgetting. This paper fills this gap by making a multi-aspect comparison of several tokenizer adaptation techniques for a fixed decoder-based LLM. In our experiments, we focus only on Russian for the sake of results clarity in the limited resources circumstances. Under controlled conditions, we compare three methods. The work establishes new baselines for tokenizer adaptation in Russian and demonstrates a computationally effective way to enhance the performance, reducing GPU hours up to 2–3 times.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Multi-Aspect Evaluation of Tokenizer Adaptation Methods for Large Language Models on Russian

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年10月1日

查看全文
  • G. D. Andriushchenko,
  • M. E. Godunova,
  • V. V. Ivanov,
  • D. S. Kuzmin,
  • A. A. Parinov,
  • A. Yu. Schenikova,
  • E. V. Zhemchuzhina

摘要

Abstract

Large language models (LLMs) pretrained on English-centered corpora have biases and perform sub-optimally on other natural languages. Adaptation of LLMs vocabulary provides a resource-efficient way to improve the quality of a pretrained model. Previously proposed adaptation techniques focus on performance (accuracy) and size metrics (fertility), ignoring other aspects in comparison, such as inference latency, compute resources for adaptation, and catastrophic forgetting. This paper fills this gap by making a multi-aspect comparison of several tokenizer adaptation techniques for a fixed decoder-based LLM. In our experiments, we focus only on Russian for the sake of results clarity in the limited resources circumstances. Under controlled conditions, we compare three methods. The work establishes new baselines for tokenizer adaptation in Russian and demonstrates a computationally effective way to enhance the performance, reducing GPU hours up to 2–3 times.