Improving Gamer Emotion Recognition with Multi-Modal Data Fusion and Ensemble Learning Techniques

These days the games are intricate and challenging, therefore, the ways of assessing gamers’ feelings must be improved. This study proposes a framework that uses facial, chats, and physiological data (heart rate) to enhance the Emotion Recognition system in games. YOLOv8 is used for real-time facial expression analysis, DistilBERT is used for in-game chat emotion analysis, and the rule-based system is employed for heart-rate data. All these modalities are combined using ensemble learning techniques particularly, bagging to provide an aggregate emotional analysis. The framework developed in the paper succeeds in attaining an overall accuracy of 99.05% in the ability to classify emotions that combine facial, text, and heart rate. Facial expression and chat analysis models, along with efficient heart rate analysis, enable permanent and accurate emotional feedback during game sessions in real time. This study is unique because this study aims to consolidate three distinct data types within a single system through complex models and data ensemble learning to facilitate a better assessment and understanding of the subject’s emotional status. It is a valuable addition to the domain of emotion detection as it provides substantial information regarding the gamer’s emotions which can be valuable in the development of better player emotion within the framework of the interactive and innovative game environment.