Silent Script: A Deep Learning Technique for Lip Reading and Dynamic Text Synthesis

Recognizing the critical role of visual lip movement in effective communication, this paper introduces an advanced lip reading system leveraging machine learning techniques. Traditional approaches to lipreading have typically split the problem into visual feature design or learning and subsequent prediction. However, recent advancements in deep lipreading, such as end-to-end trainable models, have focused strictly on word classification rather than sentence-level sequence prediction. Addressing this gap, our proposed system, named LipNet, represents a pioneering effort in end-to-end sentence-level lipreading models, simultaneously learning spatiotemporal visual features and a sequence model. Our approach integrates deep learning models, including spatiotemporal technique and SE-ResNet-18 networks, with a back-end module featuring bi-directional gated recurrent units (Bi-GRU), 1D convolution, and fully-connected layers. While existing models like spatiotemporal convolution and SE-ResNet-18 excel in certain aspects, they lack key features crucial for real-time transcription accuracy, such as capturing subtle lip movements and exploiting contextual information. By combining these models, our system greatly enhances the ability to process visual lip movement, resulting in more accurate and real-time conversion of lip movements to text.