Lip Reading Using Temporal Adaptive Module

Lip reading is a fine-grained video understanding task that endeavors to recognize speech content by analyzing the movement of the speaker’s mouth. In recent times, 3D-ResNet-18 has become the favored front-end network for most of the lip reading methods. However, a single 3D CNN layer within the 3D-ResNet-18-based front-end network might not have enough representation power to extract temporal features. To address this issue, we propose the incorporation of Temporal Adaptive Module (TAM) into the front-end network of lip reading methods. TAM is an uncomplicated temporal module that consists of two branches: a local branch that provides location-sensitive information, and a global branch that focuses on capturing long-term temporal dependencies. This combination of branches helps capture complex temporal structures and facilitates robust temporal modeling. Taking global and local relationships into consideration explicitly improves the feature representation. It can be easily used in classical building blocks of networks. We conducted ablation studies to determine the optimal TAM structure and compared our results with various related approaches on the LRW dataset. Our experimental outcomes prove the superiority of our approach.