FoCLBP+2DGF: Fusion of Complete Local Binary Pattern with 2D Geometry Features for Facial Emotion Recognition in Videos

People have different facial attributes in terms of their skin color, shape, and size. This diversity makes a challenge in the development of generalized automatic intelligent system for human’s emotions recognition through their facial expressions. Over the decades, researchers have significantly contributed to geometry and texture-based methods to act for facial contents in image/video in the form of feature vector. This paper presents a fusion of texture based complete local binary pattern (CLBP) method with two-dimensional (2D) geometry features (FoCLBP+2DGF) method to represent the facial expressions for emotion recognition in videos. The proposed method better describes the contrast and structural behavior of facial contents in spatial domain with fusion of CLBP and 2D geometry feature computing. It aggregates the facial expression’s behavior in temporal domain by sum normalization method to capture ratio among the features. The constructed final feature vector with proposed method is independent from the duration of video. Further, Bayesian optimization method is used to identify the best hyperparameters of multiclass support vector machines (SVMs) with proposed FoCLBP+2DGF method. Experimental results show that the proposed FoCLBP+2DGF method attained \(\approx\)  88.64  \(\pm\)  0.38% emotion recognition accuracy with SVMs using Gaussian kernel function in videos of crowd-sourced emotional multimodal actors’ dataset (CREMA-D). The proposed method significantly improved accuracy of intelligent system for human emotion recognition in videos over the human accuracy and existing work.