A Mixture-of-Experts Framework Based on Depth Images for Text to Video Storyboard Task
摘要
To help amateurs shoot videos or AIs generate multi-shot videos, previous works propose Text to Video Storyboard (TeViS) task that aims to generate a series of keyframes from a text synopsis. However, it is difficult for a single model to balance the relevance of keyframes to synopses and the coherence between keyframes in film languages because some input synopses are long and concrete while others are short and abstract. Moreover, most previous works are based on original images, which are not necessarily optimal for learning a TeViS model due to such images containing too much diverse details across movies. In this paper, we propose a new framework to generate storyboards based on simplified depth images by selecting appropriate expert models for different types of text synopses through a Mixture-of-Experts (MoE) architecture. First, we propose using an MoE framework by combining two expert models, one good at image-text correspondence and the other good at image-image coherence. Second, to more intelligently select the appropriate expert model, we propose combining the words in a synopsis with their Part-of-Speech (POS) as text representations. Third, we compare original images, canny images and depth images as image representations. Experimental results on publicly available datasets show that our proposed method outperforms previous methods across all metrics, e.g., \(Kendall's\, \tau \) increases by about 12.3%, and the depth images are the best representations.