Smartphones pose challenges for visually impaired users due to their reliance on touchscreens and visual feedback. While accessibility services like VoiceOver and TalkBack assist in screen navigation, they require physical contact and become cumbersome when users’ hands are occupied (e.g., holding a cane). This study proposes a non-contact alternative using electromyography (EMG) to map wrist gestures to smartphone operations. We trained a 2D convolutional neural network (CNN) on the NinaPro DB1 EMG dataset to recognize six wrist gestures corresponding to everyday touchscreen actions (e.g., screen exploration and item selection). An Android accessibility app prototype was developed to translate wrist gestures into touch inputs via ADB commands, aligning with Android Talkback accessibility logic. Preliminary user studies compare the advantages and disadvantages of touch gestures and wrist gestures, and the results show that wrist gestures have the characteristics of high usability and low learning burden. The EMG-based interaction is technically feasible as a touchless solution.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Empowering Visually Impaired Smartphone Users: A Preliminary Study on EMG-Based Smartphone Operation with Machine Learning

  • Chengxian Yin,
  • Kazushi Ahara

摘要

Smartphones pose challenges for visually impaired users due to their reliance on touchscreens and visual feedback. While accessibility services like VoiceOver and TalkBack assist in screen navigation, they require physical contact and become cumbersome when users’ hands are occupied (e.g., holding a cane). This study proposes a non-contact alternative using electromyography (EMG) to map wrist gestures to smartphone operations. We trained a 2D convolutional neural network (CNN) on the NinaPro DB1 EMG dataset to recognize six wrist gestures corresponding to everyday touchscreen actions (e.g., screen exploration and item selection). An Android accessibility app prototype was developed to translate wrist gestures into touch inputs via ADB commands, aligning with Android Talkback accessibility logic. Preliminary user studies compare the advantages and disadvantages of touch gestures and wrist gestures, and the results show that wrist gestures have the characteristics of high usability and low learning burden. The EMG-based interaction is technically feasible as a touchless solution.