In this paper, we describe a system that combines keyboard detection, key segmentation, piano transcription, and hand-tracking into a pipeline for general piano performance videos recorded from an overhead perspective. Keyboard localization was performed by a fine-tuned YOLOv8 model on manually annotated frame samples, followed by a procedure for segmenting and labeling the keys. Keyboard fingerings were predicted using a distance metric between the hands and key boundaries, achieving an 82% accuracy. Utilizing the knowledge of hand positions demonstrated up to a 6.6% improvement in piano transcription F1 score.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Towards a Dataset for Estimation of Keyboard Fingerings

  • Filip Danielsson,
  • Pavel Pecina

摘要

In this paper, we describe a system that combines keyboard detection, key segmentation, piano transcription, and hand-tracking into a pipeline for general piano performance videos recorded from an overhead perspective. Keyboard localization was performed by a fine-tuned YOLOv8 model on manually annotated frame samples, followed by a procedure for segmenting and labeling the keys. Keyboard fingerings were predicted using a distance metric between the hands and key boundaries, achieving an 82% accuracy. Utilizing the knowledge of hand positions demonstrated up to a 6.6% improvement in piano transcription F1 score.