The pronunciation of a language and corresponding lip shapes have a strong relationship. As such, technologies that align the lip movements of a sentence in the translated language with that in the original language can be useful for scenarios such as movie dubbing, particularly in singing scenes, where consistency between a character’s speech and their visual lip movements is desirable. In this paper, we define lip shape similarity based on the International Phonetic Alphabet (IPA) chart which is a knowledge on phonetics, and integrate it into a word selection algorithm for general machine translation. We propose an automatic lyric translation method that balances the semantics and the lip shape similarity when translating the source lyrics into a target language. For quantitative evaluation, by using professionally translated lyrics as a reference, we optimize the proposed method to best preserve both semantics and lip shape similarity. Experimental results demonstrate that the generated translations yield higher lip shape similarity than that by baseline translations.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Lip Shape-Aware Word Selection for Lyric Translation

  • Kotaro Ikeda,
  • Chihaya Matsuhira,
  • Hirotaka Kato,
  • Marc A. Kastner,
  • Takatsugu Hirayama,
  • Takahiro Komamizu,
  • Ichiro Ide

摘要

The pronunciation of a language and corresponding lip shapes have a strong relationship. As such, technologies that align the lip movements of a sentence in the translated language with that in the original language can be useful for scenarios such as movie dubbing, particularly in singing scenes, where consistency between a character’s speech and their visual lip movements is desirable. In this paper, we define lip shape similarity based on the International Phonetic Alphabet (IPA) chart which is a knowledge on phonetics, and integrate it into a word selection algorithm for general machine translation. We propose an automatic lyric translation method that balances the semantics and the lip shape similarity when translating the source lyrics into a target language. For quantitative evaluation, by using professionally translated lyrics as a reference, we optimize the proposed method to best preserve both semantics and lip shape similarity. Experimental results demonstrate that the generated translations yield higher lip shape similarity than that by baseline translations.