English-centric large language models (LLMs), such as LLaMA, have gained prominence in NLP research and practice. Although these models are predominantly trained on English data, their widespread adoption has prompted important attention regarding their cross-lingual generalization capabilities. While cross-lingual capabilities have been extensively explored in the context of multilingual masked language models (MMLMs), corresponding research on English-centric LLMs remains limited. However, due to their decoder-only architecture and constrained access to multilingual training data, it remains unclear whether insights gained from MMLMs apply to these English-centric models. To fill this gap, we conduct a systematic analysis of cross-lingual generalization capabilities in English-centric LLMs. Our experiments demonstrate that even when fine-tuned solely on English data, English-centric LLMs generalize across languages in both classification and generation tasks. Further analysis reveals that representation similarity to English plays a crucial role in enabling this generalization, outweighing the influence of the multilingual data ratio during pretraining. This finding contrasts with prevailing assumptions in the MMLM literature. Additionally, we propose and empirically validate a similarity-reversed data allocation strategy, one that assigns more data to languages less similar to English, which can effectively enhance overall multilingual performance, particularly under constrained data budgets.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Understanding Cross-Lingual Generalization of English-Centric LLMs: The Role of Representation Similarity and Data Exposure

  • Suchun Xie,
  • Shota Sasaki,
  • Hwichan Kim,
  • Yunmeng Li,
  • Reina Akama,
  • Jun Suzuki

摘要

English-centric large language models (LLMs), such as LLaMA, have gained prominence in NLP research and practice. Although these models are predominantly trained on English data, their widespread adoption has prompted important attention regarding their cross-lingual generalization capabilities. While cross-lingual capabilities have been extensively explored in the context of multilingual masked language models (MMLMs), corresponding research on English-centric LLMs remains limited. However, due to their decoder-only architecture and constrained access to multilingual training data, it remains unclear whether insights gained from MMLMs apply to these English-centric models. To fill this gap, we conduct a systematic analysis of cross-lingual generalization capabilities in English-centric LLMs. Our experiments demonstrate that even when fine-tuned solely on English data, English-centric LLMs generalize across languages in both classification and generation tasks. Further analysis reveals that representation similarity to English plays a crucial role in enabling this generalization, outweighing the influence of the multilingual data ratio during pretraining. This finding contrasts with prevailing assumptions in the MMLM literature. Additionally, we propose and empirically validate a similarity-reversed data allocation strategy, one that assigns more data to languages less similar to English, which can effectively enhance overall multilingual performance, particularly under constrained data budgets.