This study presents the design and implementation of a flexible translation engine tailored for automated website crawling and text chunking, with a specific application to the secondary metal market. Addressing the increasing demand for multilingual processing in Natural Language Processing (NLP), the system integrates Neural Machine Translation (NMT) and Statistical Machine Translation (SMT) with preprocessing techniques such as text chunking to enhance semantic coherence and domain-specific translation accuracy. The study evaluates various chunking and translation strategies using languages relevant to the metal industry, namely Turkish, Chinese, and Spanish, to uncover information often overlooked in English-language sources. Using crawled data from non-English news sites, the system preprocesses, segments, and translates content using models like Multilingual Bidirectional and Auto-Regressive Transformer (mBART) and Large Language Model Meta AI (LLaMA). The translation quality is evaluated using three different metrics, with a special focus on the accuracy of technical terminology. Results show that chunking enhances both translation quality and analysis accuracy by preserving contextual information, while NMT models outperform SMT in domain-specific term handling. The study proposes a transferable framework adaptable to other specialised industries, contributing valuable insights into multilingual data processing and automated market analysis.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Design and Implementation of a Flexible Translation Engine for Automated Website Crawling and Text Chunking Applied on the Metal Market as an Example

  • Nicolas Dolle,
  • Luisa Brenner,
  • Kirill Anikin,
  • Dimitrios Exarchos,
  • Marc Fernandes

摘要

This study presents the design and implementation of a flexible translation engine tailored for automated website crawling and text chunking, with a specific application to the secondary metal market. Addressing the increasing demand for multilingual processing in Natural Language Processing (NLP), the system integrates Neural Machine Translation (NMT) and Statistical Machine Translation (SMT) with preprocessing techniques such as text chunking to enhance semantic coherence and domain-specific translation accuracy. The study evaluates various chunking and translation strategies using languages relevant to the metal industry, namely Turkish, Chinese, and Spanish, to uncover information often overlooked in English-language sources. Using crawled data from non-English news sites, the system preprocesses, segments, and translates content using models like Multilingual Bidirectional and Auto-Regressive Transformer (mBART) and Large Language Model Meta AI (LLaMA). The translation quality is evaluated using three different metrics, with a special focus on the accuracy of technical terminology. Results show that chunking enhances both translation quality and analysis accuracy by preserving contextual information, while NMT models outperform SMT in domain-specific term handling. The study proposes a transferable framework adaptable to other specialised industries, contributing valuable insights into multilingual data processing and automated market analysis.