Intermediality of Musical Emotions in a Multimodal Scenario: Deep Learning-Aided EEG Correlation Study

The present work looks to study the intermediality of musical emotions from the perspective of audio-visual (AV) and audio-only (AO) stimulus and their corresponding neural manifestations. A psychological experiment was conducted on 50 non-musician participants using 8 AO and 8 AV clips representing two clips from each of the four intended emotional areas—happy, sad, calm, and anxiety, respectively. The subjects were asked to mark the appropriate emotions corresponding to each AV and AO clip and their respective intensities on a 5-point Likert scale. The average intensity of emotional elicitation from each clip was evaluated. Next, an EEG study was conducted on five participants who presented the same set of AV and AO clips as in the psychological experiment and time series data from frontal, parietal, occipital, and temporal electrodes were extracted. Convolutional Autoencoders were used to convert the AV samples to 1D time series data. A robust nonlinear cross-correlation analysis technique Multifractal Detrended Cross-Correlation Analysis (MFDXA), was then used to find the degree of cross-correlation between the EEG time series data and the 1D time series data obtained from the AO and AV clips. Thus, we have a direct correlation between the source stimuli and the output neural response, providing new information about multimodal perception and the difference between arousal-based activities related to musical emotion processing.