Foley Agent: Automatic Sound Design and Mixing Agent for Silent Videos Driven by LLMs
摘要
The current movie sound effect dubbing lacks a comprehensive automated solution. We introduce Foley Agent, an automatic movie sound design and mixing agent driven by large language models (LLMs). It processes the movie in three stages: resource collection, sound synchronization, and mixing. First, it comprehends the content of a video clip by querying a Visual LLM and searching for sound effect resources. Subsequently, it adjusts the sound signal to match the motion in the clips. Finally, it mixes the audios based on reasoning. Our approach results in structured and controllable sound with lossless quality, particularly maintaining spatial awareness in multi-channel audio. We provide insights into the capabilities of LLMs for general tasks. To the best of our knowledge, this is the first attempt to use LLMs for film dubbing and mixing.