This study aims to build a high-quality multilingual parallel corpus and optimize the performance of the translation model by improving the Transformer algorithm, which is faced with the problems of data scarcity, uneven quality, and poor translation performance of low-resource language pairs. Methodologically, a bidirectional encoding architecture based on Transformer is adopted. First, 320 million sentence pairs in 30 languages are extracted from open source data sources such as Common Crawl and TED Talks through multi-stage data cleaning and standardization. Secondly, a cross-language alignment module is designed, and the semantic space projection of low-resource languages is realized by using unsupervised word embedding mapping technology, combined with a dynamic curriculum learning strategy. Finally, a hybrid fine-tuning method is adopted to combine transfer learning with adversarial training, and the model parameter reuse rate is enhanced by adding a language-specific adapter layer. On the FLORES-101 evaluation set, the BLEU value of the low-resource language pair (Nepali-Sinhala) increased from 12.5 of the baseline model to 28.7. At the same time, the constructed corpus has achieved 99.3% sentence alignment accuracy and 95.7% vocabulary coverage after manual sampling verification. This study confirms the effectiveness of the curriculum learning strategy and hybrid fine-tuning mechanism for low-resource language translation, and provides a high-quality data foundation and model optimization solution for multilingual NLP applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multilingual Parallel Corpus Construction and Translation Quality Optimization Based on Transformer Algorithm

  • Xiaodong Wang,
  • Wanting Li,
  • Lihe Chen,
  • Lixin He

摘要

This study aims to build a high-quality multilingual parallel corpus and optimize the performance of the translation model by improving the Transformer algorithm, which is faced with the problems of data scarcity, uneven quality, and poor translation performance of low-resource language pairs. Methodologically, a bidirectional encoding architecture based on Transformer is adopted. First, 320 million sentence pairs in 30 languages are extracted from open source data sources such as Common Crawl and TED Talks through multi-stage data cleaning and standardization. Secondly, a cross-language alignment module is designed, and the semantic space projection of low-resource languages is realized by using unsupervised word embedding mapping technology, combined with a dynamic curriculum learning strategy. Finally, a hybrid fine-tuning method is adopted to combine transfer learning with adversarial training, and the model parameter reuse rate is enhanced by adding a language-specific adapter layer. On the FLORES-101 evaluation set, the BLEU value of the low-resource language pair (Nepali-Sinhala) increased from 12.5 of the baseline model to 28.7. At the same time, the constructed corpus has achieved 99.3% sentence alignment accuracy and 95.7% vocabulary coverage after manual sampling verification. This study confirms the effectiveness of the curriculum learning strategy and hybrid fine-tuning mechanism for low-resource language translation, and provides a high-quality data foundation and model optimization solution for multilingual NLP applications.