From Vision to Sound: The Application of ViT-LSTM in Music Sequence

This study combines the Visual Transformer (ViT) and Long Short-Term Memory (LSTM) networks to propose a deep learning model called ViT-LSTM, aimed at generating music sequences from music data features. We conducted this research using the MAESTRO dataset, which covers various MIDI files primarily focused on piano music. The data preprocessing involved hot encoding techniques and the creation of training and validation datasets using a sliding window strategy. We comprehensively evaluated the performance of this model using two metrics: generation diversity and generation quality. The experimental results demonstrate that ViT-LSTM can generate diverse and high quality music compared to other classical deep learning models, with a higher signal-to-noise ratio in the generated music.