Challenges of Facial Expression Recognition and Recommendations for the Use of Emotion AI in Video Conferences

Artificial emotional intelligence (AIE), affective computing or Emotion AI deal with the ability of machines to recognize human emotions. To investigate the possibilities and limitations of such technologies further, we explored an existing commercial Facial Expression Recognition (FER) tool as well as an open-source project and carried out several small-scale user studies focusing on emotion recognition from faces during video conferences. This approach aims to address the following research questions: How well can FER technologies recognize emotions in video conferences? What are the challenges and limitations of FER technologies? With this paper we contribute to the assessment and practicability of FER, present two FER tools and highlight criticism as well as limitations of FER technologies. We outline different small-scale user studies with FER. We conclude with recommendations drawn from those user studies and literature research. These suggestions adhere to principles such as Responsible AI (RAI) and value-based design. We do this by the example of FER in video conferences, but some of the findings may be transferred to other Emotion AI technologies and application areas as well.