A Study on Methods of Synchronization Between Gestural-Visual and Audiovisual Communication

The provision of a window with 3D avatars rendering the machine translation of audiovisual content into sign languages allows for a more significant offer of accessible programming, both on live television and on-demand video services, and reduces the operational costs involved, especially with interpreters. However, additional challenges involve synchronizing the original audio with the equivalent signal presentation, which is slower than human speech. Studies indicate that the presentation duration of a sign in American Sign Language (ASL) is about twice the duration of a monosyllabic word. This work aims to investigate methods and strategies to improve the synchronization of signs with the associated video content. One of the approaches considered is the application of automatic sentence summarization (or compression), which aims to produce a shorter paraphrase for a given input sentence in a simplification process to require fewer signs to convey the original content. Another applicable technique is called elastic adjustment, which consists of modifying the display duration of a media object. Elastic tuning algorithms can adjust the audio speed by up to 10%, up or down, keeping the audio perception quality within acceptable limits. The primary hypothesis is that combining such methods reduces the time required for interpreting content in Brazilian Sign Language, thereby reducing the information desynchronization and prioritizing the most crucial information related to that content. Two experiments with automatic summarization of a news article were conducted. In both, it was possible to display the signing in Brazilian Sign Language using a 3D avatar within the duration of the news segment. The first experiment had a word summarization rate of 31.5%, and the second had a rate of 56.5%.