CNN Based Real Time Detection of Words from Lip Movements and Automated into Text

Recently developed techniques for information extraction from visual footage without audio or distorted audio have been made possible by developments in computer vision. The use of analyzing lip movements to produce and recognize text or speech is one such field of study. This method can be especially helpful when audio material is either missing or of poor quality. This paper suggests a Convolutional Neural Network (CNN) based technique for automating the conversion of lip movements into text and conducting a comparison with existing approaches. The method starts with videos of an individual uttering specific words or phrases and assembles them into a dataset at random. Each frame’s lip region is isolated as a Region of Interest (ROI), extracted consecutively, and categorically labeled in accordance with the spoken word or phrase in the corresponding videos. A 2D-CNN model was specifically designed to train upon this dataset as a multi-class classification problem and further introduced a prediction model to take video input and iterate through all the frames using the trained model to determine which word or phrase has the highest probability of being uttered in the video. Experimental findings show that this approach outperforms the majority of the previously presented conventional complicated approaches, achieving an acceptable accuracy of 84.7%. In particular, when audio data is either missing or of poor quality, this technique provides a promising solution for generating and recognizing text or speech from visual footage. The creation of the dataset, training of the CNN model, and application of the prediction model are all covered in depth in this thesis, along with a thorough analysis of the outcomes.