Analysis of Lombard Effect by Using Hybrid Visual Features for ASR

In this work, a lip movements related hybrid visual feature is proposed to analyze the Lombard effect in automatic speech recognition task. First, a robust lip contour method is employed to automatically detect the lip contours. Three vertical heights and width of the lip contour are considered as geometric features. By using Lucas-Kanade algorithm, the vertical velocity and horizontal velocity are estimated from eight points (end point of the vertical heights and width of the lip contour) between every successive frames. These motion parameters are concatenated to geometric lip features, results a hybrid visual feature. As compared to geometric features, the proposed hybrid visual feature improves the word recognition accuracy by 7.5%, reflecting the benefit of fusing the motion information. Later, the proposed hybrid visual feature is used to investigate the Lombard effect in speech recognition task. The experimental studies are made with Audio-Visual Lombard Grid database. It is observed that vertical and horizontal velocities of Lombard speech are comparatively more than the plain speech. Further, the influence of Lombard effect is observed in mismatched condition. This can be countered by including Lombard train data while building audio visual speech recognition system.