Scholarlink

Large language models have transformed natural language processing, but their effectiveness is limited for regional language variants due to underrepresentation in training data. This paper introduces Cecilia 2B, a 2-billion-parameter language model continually pretrained on nearly 1 billion tokens of Cuban Spanish text, including newspapers, encyclopedias, legal documents, literature, and song lyrics, to address the gap in language technology for Cuban Spanish. Leveraging the Salamandra 2B architecture, Cecilia 2B demonstrates the feasibility and value of adapting multilingual models to regional variants through continual pretraining, achieving improved performance on Spanish and multilingual tasks relevant to its target domain while maintaining computational efficiency suitable for resource-constrained environments. We detail the construction of a culturally rich Cuban Spanish corpus, the adaptation methodology, and a comparative evaluation with the base model, highlighting both the benefits and trade-offs of regional specialization. Cecilia 2B provides a foundational resource for Cuban Spanish NLP and establishes a path for future research in instruction tuning, corpus expansion, tokenizer retraining, and the development of larger or more specialized models.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Continual Pretraining of a Small Language Model on Cuban Spanish Corpora

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Ernesto Luis Estevanell-Valladares,
  • Suilan Estevez-Velarde,
  • Alejandro Piad-Morffis,
  • Yudivian Almeida-Cruz,
  • Alejandro Beltrán Varela,
  • Carla Sunami Pérez Valera,
  • Daniel Alejandro Valdés Pérez,
  • Deborah Famadas Rodríguez,
  • Elena Rodríguez Horta,
  • Gabriel Hernández Rodríguez,
  • Niley González Ferrales,
  • Roberto Garcia Rodriguez,
  • Roberto Marti Cedeño,
  • Juan Pablo Consuegra Ayala,
  • Robiert Sepúlveda-Torres,
  • Yoan Gutiérrez,
  • Andrés Montoyo,
  • Rafael Muñoz Guillena,
  • Manuel Palomar

摘要

Large language models have transformed natural language processing, but their effectiveness is limited for regional language variants due to underrepresentation in training data. This paper introduces Cecilia 2B, a 2-billion-parameter language model continually pretrained on nearly 1 billion tokens of Cuban Spanish text, including newspapers, encyclopedias, legal documents, literature, and song lyrics, to address the gap in language technology for Cuban Spanish. Leveraging the Salamandra 2B architecture, Cecilia 2B demonstrates the feasibility and value of adapting multilingual models to regional variants through continual pretraining, achieving improved performance on Spanish and multilingual tasks relevant to its target domain while maintaining computational efficiency suitable for resource-constrained environments. We detail the construction of a culturally rich Cuban Spanish corpus, the adaptation methodology, and a comparative evaluation with the base model, highlighting both the benefits and trade-offs of regional specialization. Cecilia 2B provides a foundational resource for Cuban Spanish NLP and establishes a path for future research in instruction tuning, corpus expansion, tokenizer retraining, and the development of larger or more specialized models.