Representation and Generation of Music: Incorporating Composers’ Perspectives into Deep Learning Models

The art nature of music makes it difficult, if not impossible, to extract solid rules from composed pieces and express them mathematically. This has led to the lack of utilization of music expert knowledge in the AI literature for the automation of music composition. In this study, we employ intervals, which are the building blocks of music, to represent musical data closer to human composers’ perspectives. Based on intervals, we develop and train OrchNet which translates musical data into numerical vector representation. Furthermore, another model called CompoNet is developed and trained to generate music. Using intervals and a novel monitor-and-inject mechanism, we address the two main limitations of the existing literature: lack of orchestration and lack of long-term memory. The music generated by CompoNet is evaluated in a human-subject study: whether human judges can tell the difference between the music pieces composed by humans versus those generated by our system. The data is analyzed using MannWhitney U Test, and the results show no statistically significant difference between human-composed music versus what our system generated.