<p>Sign language is an essential communication tool for individuals with hearing and speech impairments, who often face significant challenges in interacting with others in their daily lives. This study addresses these challenges by developing a recognition system for Turkish Sign Language (TSL) tailored to healthcare settings. To achieve this, surface electromyography (sEMG) and inertial measurement unit (IMU) signals are collected from the Myo armband during dynamic TSL gestures. A dataset is created with recordings from 19 participants performing word-based and sentence-based gestures. The recorded signals are transformed into 2D images—sEMG signals via direct channel summation and IMU (Gyro) signals via 2D Mel spectrogram—and classified using five variants of the YOLOv8 model. In the subject-dependent analysis, the highest classification accuracy for IMU signals is achieved by YOLOv8s and YOLOv8m on the word-based dataset (0.85) and by YOLOv8x on the sentence-based dataset (0.85), while sEMG-based classification accuracy reaches up to 0.76 (YOLOv8n, word-based) and 0.84 (YOLOv8l, sentence-based). For the combined word–sentence dataset, IMU-based classification accuracy achieves up to 0.85 under YOLOv8x and sEMG-based classification accuracy reaches up to 0.75 under YOLOv8x and YOLOv8m. For subject-independent evaluation using the Leave-One-Subject-Out (LOSO), ResNet18 and EfficientNet-B0 achieve the highest Top-1 accuracies—up to 0.94 for IMU and 0.88 for sEMG signals—outperforming YOLOv8m and the raw signal-based 1D-CNN baseline across word-based, sentence-based, and combined datasets. This study provides a detailed evaluation of sEMG and IMU as separate inputs for gesture recognition and demonstrates the potential of Convolutional Neural Network-based to address real-world communication challenges faced by TSL users.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Convolutional neural network-based Turkish sign language recognition using sEMG and IMU signals

  • Kübra Erat,
  • Beytullah Ahmet Kından,
  • Orhan Akbulut,
  • Pınar Onay Durdu

摘要

Sign language is an essential communication tool for individuals with hearing and speech impairments, who often face significant challenges in interacting with others in their daily lives. This study addresses these challenges by developing a recognition system for Turkish Sign Language (TSL) tailored to healthcare settings. To achieve this, surface electromyography (sEMG) and inertial measurement unit (IMU) signals are collected from the Myo armband during dynamic TSL gestures. A dataset is created with recordings from 19 participants performing word-based and sentence-based gestures. The recorded signals are transformed into 2D images—sEMG signals via direct channel summation and IMU (Gyro) signals via 2D Mel spectrogram—and classified using five variants of the YOLOv8 model. In the subject-dependent analysis, the highest classification accuracy for IMU signals is achieved by YOLOv8s and YOLOv8m on the word-based dataset (0.85) and by YOLOv8x on the sentence-based dataset (0.85), while sEMG-based classification accuracy reaches up to 0.76 (YOLOv8n, word-based) and 0.84 (YOLOv8l, sentence-based). For the combined word–sentence dataset, IMU-based classification accuracy achieves up to 0.85 under YOLOv8x and sEMG-based classification accuracy reaches up to 0.75 under YOLOv8x and YOLOv8m. For subject-independent evaluation using the Leave-One-Subject-Out (LOSO), ResNet18 and EfficientNet-B0 achieve the highest Top-1 accuracies—up to 0.94 for IMU and 0.88 for sEMG signals—outperforming YOLOv8m and the raw signal-based 1D-CNN baseline across word-based, sentence-based, and combined datasets. This study provides a detailed evaluation of sEMG and IMU as separate inputs for gesture recognition and demonstrates the potential of Convolutional Neural Network-based to address real-world communication challenges faced by TSL users.