A Systematic Review of Multilingual and Multimodal Speech Recognition: Integrating ASR, VSR, and AVSR with Self-Supervised Learning
摘要
Speech recognition development has become a rapidly developing technique, not based on statistical techniques. In this paper, recent developments in speech recognition, deep learning, and, more recently, self-supervised methods are discussed. Although audio speech recognition (ASR) has achieved a state-of-the-art level in various languages, it remains vulnerable in the presence of noise. Visual speech recognition (VSR) utilizes lip action, but it is not Accurate. Audiovisual speech recognition (AVSR) is a form of speech recognition that attempts to enhance its accuracy by combining modalities. Current models are primarily unable to be considered multi-modal or multilingual, rendering them ineffective in the context of the real world, which is increasingly multilingual and multi-modal. This Research paper is a systematic review of the use of ASR, VSR, and AVSR, with a particular focus on self-supervised learning (SSL) frameworks, which enable the utilization of unlabeled large-scale audiovisual data. To analyze recent approaches, including HuBERT, AV-HuBERT, Data 2Vec, AV-Data 2Vec, XLS-R, BRAVEn, Unified Speech Recognition (USR), and Matryoshka-based models of multi-modal approaches, we discuss their architectures, datasets, evaluation benchmarks, and performance trends. The key research gaps are as follows: the absence of coherent, multilingual, and multi-modal models; the facilitation of code-switching, cross-lingual transfer, and the trade-off between efficiency and resources available in deployed applications. The review identifies new opportunities in unified architectures, cross-lingual transfer of CSS, adaptive multi-modal fusion strategies, and connections with large language models to offer context-related recognition. With a blend of theoretical and practical illustrations, the paper provides guidelines on how to make multilingual, multi-modal speech recognition systems universal, robust, and scalable, so that they can be easily accessible, enhance the human-machine interface, and be practically implemented.