PainFusion: Multimodal Pain Assessment from RGB and Sensor Data

Traditional pain assessment tools often rely on subjective self-reporting methods, hindering the work of healthcare professionals. However, the patient’s facial expressions and biomedical data provide a reliable source of information for caregivers. In this work, we present a multimodal architecture that utilizes both RGB video and biomedical sensor data from the BioVid Heat Pain dataset. We use video transformer architectures in conjunction with a thorough analysis of biomedical signals, including galvanic skin response, electromyography, and electrocardiogram, for comprehensive feature extraction. These features are then fused to create a robust model for pain assessment. Experimental results show that our multimodal architecture outperforms unimodal video-based methods in pain detection. Furthermore, our study highlights the potential of combining non-invasive video analysis with physiological data to facilitate pain prediction and management in clinical settings, paving the way for more accurate and efficient pain assessment methods that can be used in various healthcare applications.