With the rise of multilingual technology and the growing demand for inclusive language tools, there remains a significant gap when it comes to resources and research dedicated to Assamese, one of the major languages spoken in North east India. The core objective is to bridge that gap by leveraging recent advances in deep learning—specifically the Transformer architecture—to build a translation model that is both accurate and adaptable. The system is implemented using the OpenNMT-py framework which supports flexible training of encoder-decoder models with attention mechanisms. To train the model, a parallel corpus consisting of approximately 124,000 English-Assamese sentence pairs was sourced from the Samanantar dataset available on Kaggle. The data was prepared using a machine translation (MT) preprocessing script, which utilizes Sentence Piece to perform sub word tokenization. This step ensures that rare and compound words are broken down into more frequently occurring subunits, thereby improving the model’s ability to learn patterns in low-resource language settings. The trained model was evaluated using the BLEU score, which is used to compare the overlap between machine-generated outputs and human reference translations. The evaluation results indicated that the Transformer-based approach provides promising performance, particularly in producing fluent and contextually relevant translations, despite the limited size of the dataset.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Development of English-Assamese Machine Translation System Using NMT

  • Saptarshi Paul,
  • Pratul Kalita,
  • Abu Ahmed Waliullah Mazumder,
  • Saurav Paul

摘要

With the rise of multilingual technology and the growing demand for inclusive language tools, there remains a significant gap when it comes to resources and research dedicated to Assamese, one of the major languages spoken in North east India. The core objective is to bridge that gap by leveraging recent advances in deep learning—specifically the Transformer architecture—to build a translation model that is both accurate and adaptable. The system is implemented using the OpenNMT-py framework which supports flexible training of encoder-decoder models with attention mechanisms. To train the model, a parallel corpus consisting of approximately 124,000 English-Assamese sentence pairs was sourced from the Samanantar dataset available on Kaggle. The data was prepared using a machine translation (MT) preprocessing script, which utilizes Sentence Piece to perform sub word tokenization. This step ensures that rare and compound words are broken down into more frequently occurring subunits, thereby improving the model’s ability to learn patterns in low-resource language settings. The trained model was evaluated using the BLEU score, which is used to compare the overlap between machine-generated outputs and human reference translations. The evaluation results indicated that the Transformer-based approach provides promising performance, particularly in producing fluent and contextually relevant translations, despite the limited size of the dataset.