Scholarlink

<p>Multimodal Machine Translation (MMT) aims to improve translation quality by combining textual input with additional modalities such as visual context. In recent years, several promising MMT approaches have been proposed for various languages, including Indian languages. However, most of these methods struggle to accurately translate emotional or ambiguous expressions, especially in under-resourced languages like Hindi. While datasets like Hindi Visual Genome (HVG) support English-Hindi MMT, they often lack emotional depth and fail to capture real-world ambiguity. To address this gap, this research introduces the SentiCap-HIN Dataset, derived from SentiCap. It pairs English-Hindi image captions to better handle linguistic diversity and ambiguity. This dataset serves as a valuable benchmark for developing visually grounded and semantically sensitive translation models for Indian languages. The study also proposes a novel vision-aware MMT framework called MMT-CMAI-T (Cross-Modal Attention Injection with Structured FeatureTrie) to integrate image features with text using a structured attention mechanism. It employs attention entropy to selectively inject relevant visual information, organising missed semantic features during translation. This enables context-aware querying of visual memory based on the decoder’s evolving state. Experimental results show that MMT-CMAI-T significantly outperforms existing MMT and text-only baselines on the SentiCap-HIN corpus, using BLEU, chrF, and METEOR metrics. The research marks progress toward more adaptive, interpretable, and culturally aware translation systems that effectively use multimodal context while preserving linguistic structure. Significance Machine translation systems often struggle with ambiguous words, resulting in inaccurate translations that impede cross-cultural communication. This research presents a novel method that combines visual context with text, allowing translation models to clarify meanings in a way that resembles natural human understanding. The SentiCap-HIN dataset and MMT-CMAI-T framework specifically tackle the difficulties of English-to-Hindi translation, where lexical ambiguity poses a significant challenge. This approach provides notable benefits, including enhanced translation precision, improved management of context-sensitive expressions, and increased accessibility for millions of Hindi speakers engaging with global digital content.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Cross-Modal Attention with CMAI-T: Improving Multimodal Translation for Hindi Using the SentiCap-HIN Dataset

  • 期刊论文

  • 发表日期：

    2026年4月6日

查看全文
  • Binnu Paul,
  • Dwijen Rudrapal,
  • Kunal Chakma,
  • Anupam Jamatia,
  • Rishabh Chaudhary

摘要

Multimodal Machine Translation (MMT) aims to improve translation quality by combining textual input with additional modalities such as visual context. In recent years, several promising MMT approaches have been proposed for various languages, including Indian languages. However, most of these methods struggle to accurately translate emotional or ambiguous expressions, especially in under-resourced languages like Hindi. While datasets like Hindi Visual Genome (HVG) support English-Hindi MMT, they often lack emotional depth and fail to capture real-world ambiguity. To address this gap, this research introduces the SentiCap-HIN Dataset, derived from SentiCap. It pairs English-Hindi image captions to better handle linguistic diversity and ambiguity. This dataset serves as a valuable benchmark for developing visually grounded and semantically sensitive translation models for Indian languages. The study also proposes a novel vision-aware MMT framework called MMT-CMAI-T (Cross-Modal Attention Injection with Structured FeatureTrie) to integrate image features with text using a structured attention mechanism. It employs attention entropy to selectively inject relevant visual information, organising missed semantic features during translation. This enables context-aware querying of visual memory based on the decoder’s evolving state. Experimental results show that MMT-CMAI-T significantly outperforms existing MMT and text-only baselines on the SentiCap-HIN corpus, using BLEU, chrF, and METEOR metrics. The research marks progress toward more adaptive, interpretable, and culturally aware translation systems that effectively use multimodal context while preserving linguistic structure. Significance Machine translation systems often struggle with ambiguous words, resulting in inaccurate translations that impede cross-cultural communication. This research presents a novel method that combines visual context with text, allowing translation models to clarify meanings in a way that resembles natural human understanding. The SentiCap-HIN dataset and MMT-CMAI-T framework specifically tackle the difficulties of English-to-Hindi translation, where lexical ambiguity poses a significant challenge. This approach provides notable benefits, including enhanced translation precision, improved management of context-sensitive expressions, and increased accessibility for millions of Hindi speakers engaging with global digital content.