Exploring Physiology-Based Classification of Flow During Musical Improvisation in Mixed Reality

The flow state is desirable in many activities, e.g., while making music or being active in Virtual, Augmented, and Mixed Realities. With the long-term goal of creating affective systems that can consider the user’s flow state in real-time, we evaluated an approach for real-time flow classification during networked music performance using a deep neural network. We trained our classifier based on physiological signals (PPG and GSR) that we recorded and annotated in a laboratory study, including jamming musicians. The results that we present in this paper confirm the technical validity of this approach while also facing challenges that stem from inter-rater reliability and a heavily unbalanced dataset.