Emotion and Instrument Recognition from Indian Classical Music Recordings Using Transformers

Since the inception of Deep Neural networks, research in the field of sequence modeling and sequence transduction has advanced rapidly, based on complex recurrent/convolutional networks and encoder-decoder networks. Transformer networks perform robustly across a wide variety of language modeling tasks, from Machine Translation to text classification and generation, by learning powerful representations from speech with the help of attention mechanisms and are the current state of the art for text sequence modeling. Recent advancements have shown these to be equally robust for learning powerful representations from audio data to get better results on speech-to-text conversion tasks and classification tasks like Speech Emotion Recognition (SER). However learning representations from audio data of Indian Classical Music (ICM) instruments and using the models to classify instrument types and emotions is an interesting problem that has recently started gathering attention. We have used the transformer wav2vec2.0 architecture to learn representations from audio samples of Indian classical music and classify them based on the instruments present in the examples and their emotional content. Labeled training data of three instruments, namely, the flute, sitar, and sarod, pre-classified into four emotions, anxiety, sadness, happiness, and calm, were used to train a Transformer Network, outperforming traditional approaches of pattern identification which used extracted audio features for machine learning or other representation based deep learning methods using only convolutional or recurrent architectures. The results show that the architecture used on the ICM dataset learns sequential representation from the samples and classifies the data depending on the instruments present and the positive (happy and calm) and negative (sad and anxiety) emotions on the valence scale with high accuracy.