MusicGAIL: A Generative Adversarial Imitation Learning Approach for Music Generation

Deep learning based automatic music generation has received significantly attention and became an attractive research topic in recent years. However, most existing methods face three major challenges: (1) Models based on the Recurrent Neural Networks (RNNs) trained by Maximum Likelihood Estimation (MLE) suffer from the exposure bias problem, while some Generative Adversarial Networks (GANs) based methods have been proposed to alleviate it, they still suffer from the problems of reward sparsity and mode collapse. (2) Deep Reinforcement Learning (DRL) based models encounter the reward sparsity problem and it is impossible to manually specify a completely reasonable reward function. (3) Although Transformer can be used as an alternative to show the better parallelization and performance over the origin RNNs, it is still extremely difficult to optimize for the standard transformer structure in RL setting. In this paper, we propose MusicGAIL, a generative model for generating the online countermelody for folk melodies under the framework of generative adversarial imitation learning (GAIL) that directly learn the optimal policy from dataset without laboriously defining the reward. As the standard Transformer is hard to be applied in RL setting, we adopt Gated Transformer-XL (GTrXL) to stabilize the model training. Experimental results show that the melody pieces generated from MusicGAIL achieve better quality and diversity. Moreover, their subjective scores significantly higher than baselines in terms of melody, harmony, rhythm, folk style and emotion.