Music Scoring Based on Piano Performance Videos and Sound Sources

In recent years, with the spread of social media, performance videos with arrangements have been attracting attention, and the demand for musical scores has been increasing. However, performers themselves rarely create music notation, and it is very difficult for viewers to imitate advanced performances. Therefore, music notation created from performance videos can satisfy this demand. So far, automatic music transcription (AMT) has been performed by methods based on visual information and methods based on sound sources. However, it is difficult to estimate using only one type of information, and when people actually perform music transcription, it is common to use both visual information and sound sources. If the source information is not accurate, then there will be deviations in the music notation process, so greater accuracy is required. This research focuses on melody recognition, which is necessary for music notation, and aims to compare and evaluate multiple methods. Melody recognition involves estimating the pitch and the time corresponding to the pitch. An upright piano is used for filming, with the piano lid opened, and a video of the piano hammer taken from directly above is used.