Sequence Modeling Based Data Augmentation for Micro-expression Recognition

Micro-expressions (MEs) can reveal people’s true emotions and expose deceitful behaviors. With the introduction of deep learning, the accuracy of micro-expression recognition (MER) has been greatly improved. However, limited by insufficient and unbalanced ME samples, deep models are likely to suffer from overfitting and are easy to lean towards majority classes, resulting in unsatisfactory recognition performance. In this paper, we propose a novel sequence modeling based data augmentation (SM-DA) method to enrich the limited training samples. Specifically, we model ME sequences for a static face by remapping the motion information of an original ME sequence to it, thus synthesize new ME sequence. With current manually-annotated MEs and a large amount of publicly available static faces, the scale and diversity of ME samples could be greatly increased. Besides, to overcome the shortcoming of class-imbalance, we propose a balanced loss function, in which the loss of each category is weighted by a factor determined by both the actual sample size and effective sample size of this category. Experimental results on three benchmark ME databases demonstrate the superiority of our approach over other state-of-the-art methods.