Lip Shape-Aware Word Selection for Lyric Translation

The pronunciation of a language and corresponding lip shapes have a strong relationship. As such, technologies that align the lip movements of a sentence in the translated language with that in the original language can be useful for scenarios such as movie dubbing, particularly in singing scenes, where consistency between a character’s speech and their visual lip movements is desirable. In this paper, we define lip shape similarity based on the International Phonetic Alphabet (IPA) chart which is a knowledge on phonetics, and integrate it into a word selection algorithm for general machine translation. We propose an automatic lyric translation method that balances the semantics and the lip shape similarity when translating the source lyrics into a target language. For quantitative evaluation, by using professionally translated lyrics as a reference, we optimize the proposed method to best preserve both semantics and lip shape similarity. Experimental results demonstrate that the generated translations yield higher lip shape similarity than that by baseline translations.