TFS Recognition Using MediaPipe Hands
摘要
Thai Finger Spelling (TFS) is an essential part of Thai sign language, adopted nationwide in deaf education and community in Thailand. TFS is for a spelling part of the language and is used for spelling out proper names. To handle a large set of Thai alphabets, TFS employs various signing schemes. Some of TFS schemes, i.e., single-hand schemes, have been actively studied in previous studies, while others, i.e., point-on-hand schemes, are largely unexplored. Point-on-hand (PoH) signings rely on precise localization of key landmarks on a palm and fingers. This nature of PoH schemes demands a more fine-grained approach than a classifier-based method, widely used for single-hand schemes. This article examines application of MediaPipe Hands (MPH) to an automatic recognition of signings in TFS, particularly PoH signings. Our work studies effectiveness and limitations of MPH under the context of TFS sign recognition. For PoH sign inference on half-body images with natural background, our best-performing MPH has reached 60.4% accuracy, conferring to 42.95% accuracy shown using a trained VGG classifier. Our findings reveal MPH limitations particularly on handling hand-hand interaction (contributing to 32.8% error). Since PoH signings involve a high degree of hand-hand interaction (average 4.37%; max 89.88%), MPH in its current state is not recommended for addressing PoH sign recognition. In addition, %palm-overlapping is proposed to quantify a degree of hand-hand interaction.