L-VOCAL: Language-based Video Colorization with Audio Alignment
摘要
While language-based video colorization addresses the inherent ambiguity of color assignment, language descriptions typically focus on central objects, neglecting the crucial context of emotional tone and surrounding environment necessary for accurate film colorization. In this paper, we introduce L-VOCAL, a novel framework for language-based video colorization that leverages audio alignment to supplement context not explicitly provided by language. L-VOCAL pretrains an alignment model to establish correspondences between color and audio, enabling the learning of emotional tone and environmental atmosphere. Subsequently, these aligned audio features guide the colorization process through specially designed condition injection modules. We additionally contribute L-VACOLOR, a new dataset tailored for this task, consisting of cinematic clips with diverse color and audio tones for training and evaluation. Extensive experimental results demonstrate that L-VOCAL produces colorization results that more accurately reflect filmmakers’ artistic expression.