Real-Time Emotion Recognition in Online Video Conferences for Medical Consultations

In this paper, we present a novel solution for real-time emotion analysis in video conferencing, aiming to enhance therapies for both patients and therapists. Building upon the SenseCare project, we extend its capabilities to include a video conferencing platform with scalable real-time emotion analysis using widely available software and frameworks avoiding critical vendor locks. Our architecture focuses on ease of adaptation and further development, connecting a WebRTC conferencing platform to a scalable Kubernetes backend for emotion analysis. Emphasizing low latency, we implement the producer-consumer pattern and utilize a message broker. For emotion analysis, we use convolution neural networks. We propose a methodology for identifying an optimal batch size that maximizes backend efficiency while maintaining low latency. Our approach exhibits scalability, allowing for seamless adaptation during periods of high system utilization. Our findings demonstrate the feasibility of employing CNNs for sub-second emotion analysis on an affordable Kubernetes cluster, enabling multiple users to effectively engage in the system as patients and therapists.