An Audio-Visual Speech Separation and Personalized Keyphrase Detection in Noisy Environments
摘要
Humans possess an exceptional ability to focus on a specific audio source amidst noise, known as the cocktail party effect. Inspired by this, our project focuses on creating a system that isolates audio from individual speakers in a multi-speaker environment using advanced attention models. By leveraging RMS and Pearson correlation coefficients, the system effectively relates variations in audio signals to corresponding lip movement energy. This accurate mapping ensures better synchronization of audio with the respective speaker. Additionally, we are generating personalized captions for each individual speaker, further enhancing the accessibility and clarity of multi-speaker content. The system demonstrated reliable performance in separating speakers’ audio even in moderately noisy environments, paving the way for applications in accessibility tools and content processing.