Examining the Relationship Between Playing a Chord with Expressions and Hand Movements Using MediaPipe

This paper examined whether hand movements are responsible for expression in a piano performance. A player played a chord 100 times each with 12 different performance expressions, which consisted of three articulations (tenuto, heavy staccato, and light staccato) and four-level dynamics. The landmarks’ coordinates of her right fingers, wrist, elbow, and shoulder estimated as she played the chord (12 \(\times \) 100 times), judged by MediaPipe Pose and Hands, were used for machine learning training and testing. In the results for the learning model, the testing accuracy rate was 0.99. In each performance expression, F1-scores were 0.94–1.00. This suggested a relationship between performance expressions and hand movements. Moreover, when the player happened to play a different, unintended type of expression, her landmarks’ coordinates were close to those when she had aimed exactly to play that type of performance expression.