Hand gesture recognition and facial expression analysis are often treated as separate research problems in the existing literature. Although hand-over-face occlusions are commonly regarded as a limitation in facial expression recognition, they frequently serve as cues for mental states. In spontaneous conversations, hands over the face are frequent and informative about the participants’ mental states, providing essential cues for understanding non-verbal communication. This work systematically explores various hand-over-face gestures using recent machine learning algorithms. We establish a dataset comprising 30 hand-over-face gestures in laboratory settings, featuring 15 subjects mimicking subconscious scenarios. In our investigation, we propose a novel two-stage approach for hand-over-face gesture classification. Initially, an encoder is trained using contrastive learning, and subsequently, the trained encoder serves as a feature extractor for a feature-driven transformer classifier. This unique methodology explores the potential of contrastive learning for acquiring discriminative features for hand-over-face gesture classification. To the best of our knowledge, our proposed model is the first to investigate contrastive learning for the hand-over-face gesture classification task. Our approach achieves new state-of-the-art results on the provided dataset, surpassing recent image classification methods in the literature, with an impressive accuracy of 88.68%. Furthermore, we evaluate the extensibility of our method on the widely-used facial expression dataset FER2013 and RAFDB, achieving a comparable performance of 71.30% and 85.74% on the test set, even though the proposed model is not specifically tailored to the facial expression recognition problem.