Lip Reading Filipino Words Using Convolutional Neural Networks and Recurrent Neural Networks

Lip reading is a skill that allows a person to recognize words and sentences through the movement of the mouth. The boom of audio-video data, particularly in teleconferencing and streaming platforms, make lip reading an interesting area of research particularly in low resource languages. Using ten Filipino words, the researchers collected a video dataset of Filipinos speaking those words to train and experiment with three different configurations of deep learning models namely Inception v3 CNN and GRU model, VGG-19 CNN and GRU model, and GRU model to lip read Filipino words and to determine each configuration’s accuracy. Different combinations of parameters were used for each model, along with both original and augmented datasets. The study determined that apart from the low volume of training data, the features extracted by the CNN were not sufficient to predict the word being spoken. Furthermore, performing data augmentation did not significantly increase the model’s accuracy. Training using the height of the lip’s opening as a feature on the GRU model produced the best results—achieving a test accuracy of 17% and a validation accuracy of 26%.