Music generation is a complex and challenging problem that has made significant progress through recent machine learning solutions. The challenge lies not only in rendering natural-sounding audio but also in capturing the underlying musical structure. This paper presents a real-time system that generates a musical accompaniment for an input lead melody in the MIDI format. In this paper, we introduce REMIBlock, a novel tokenization approach, and show that it is suited for accompaniment generation. Our method uses the GPT-2 architecture [11], optimized for efficient on-device performance, to generate accompaniments that are rhythmically and harmonically coherent with a musician’s performance. The model used in this papers reaches a perplexity of 7.69 on the test set, proving the model’s ability to understand musical language. Additionally, using two metrics—groove and scale consistency—proposed in [3], we show that the generated accompaniments closely match the ground truth, differing by at most 3.4%. This further highlights our model’s ability to capture the intricacies of musical language. The model weights and inference code is available at https://github.com/UncleBen420/JazzyGPT2 .