Humans possess an exceptional ability to focus on a specific audio source amidst noise, known as the cocktail party effect. Inspired by this, our project focuses on creating a system that isolates audio from individual speakers in a multi-speaker environment using advanced attention models. By leveraging RMS and Pearson correlation coefficients, the system effectively relates variations in audio signals to corresponding lip movement energy. This accurate mapping ensures better synchronization of audio with the respective speaker. Additionally, we are generating personalized captions for each individual speaker, further enhancing the accessibility and clarity of multi-speaker content. The system demonstrated reliable performance in separating speakers’ audio even in moderately noisy environments, paving the way for applications in accessibility tools and content processing.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An Audio-Visual Speech Separation and Personalized Keyphrase Detection in Noisy Environments

  • Srajan Kumar,
  • B. Nayak Tejas,
  • Hithaish,
  • Rishab R. Budale,
  • Priya R. Kamath

摘要

Humans possess an exceptional ability to focus on a specific audio source amidst noise, known as the cocktail party effect. Inspired by this, our project focuses on creating a system that isolates audio from individual speakers in a multi-speaker environment using advanced attention models. By leveraging RMS and Pearson correlation coefficients, the system effectively relates variations in audio signals to corresponding lip movement energy. This accurate mapping ensures better synchronization of audio with the respective speaker. Additionally, we are generating personalized captions for each individual speaker, further enhancing the accessibility and clarity of multi-speaker content. The system demonstrated reliable performance in separating speakers’ audio even in moderately noisy environments, paving the way for applications in accessibility tools and content processing.