Towards End-to-End Text to Sign Language Video with Diffusion Models
摘要
In this study, we introduce TuneSL, a method that lays the foundation for direct text-to-sign language video translation, addressing challenges related to structure, coherence, and spatio-temporal precision in sign languages (SLs) to a certain degree. Unlike previous SL production (SLP) approaches relying on skeletal poses, TuneSL is the first end-to-end model, eliminating intermediary steps. Our methodology leverages a state-of-the-art text-to-video (T2V) model pipeline, incorporating a pre-trained text-to-image (T2I) diffusion model. Fine-tuning of the T2V model is performed on the How2Sign dataset. The output comprises 20-frame videos, with training using uniformly sampled frames from dataset videos, each containing up to 150 frames. Quantitative assessments show TuneSL’s superior visual quality over GAN models with skeletal pose guidance. Qualitatively, the model moderately captures SL nuances, including facial expressions. Our novel inference process involves repeating a single frame 20 times, applying diffusion and DDIM inversion, and using the inverted latent as initial noise to provide structure guidance for facilitating SL video translations for any input prompt.