Scholarlink

<p>Pakistan Sign Language (PSL) is the primary mode of communication for the Deaf community in Pakistan, yet research in PSL recognition remains underdeveloped. The primary challenge lies in the limitations of its dataset, the PSL Dictionary, which contains only two samples per sign, making it difficult to develop accurate recognition systems. While the action recognition models achieve high accuracy, they are resource-intensive, requiring substantial storage and computational power. To address these challenges, we propose a novel modality transformation approach that converts video-based sign language gestures into single images, allowing the use of image recognition models instead. In our approach, 21 key landmarks are first extracted from each video using MediaPipe. The movement of each landmark, such as the index fingertip, is tracked across all frames and represented as a single image per class, resulting in 21 distinct feature representations per video sample. To further enhance recognition performance, we introduce feature aggregation as part of this transformation. During feature aggregation multiple landmark images of the same class were combined into a single denser image. This integrated pipeline of modality transformation and feature aggregation allows the model to learn richer spatial representations from limited data. This reduces both storage and computational requirements while maintaining comparable performance to the action recognition models. We utilized Convolutional Neural Networks (CNN) to develop an efficient image recognition-based PSL recognition system. Although CNNs require a lot of training data, our model achieved superior performance in PSL recognition compared to existing techniques. Finally, building on our modality transformation approach, we also demonstrated the effectiveness of feature-specific ensemble learning, which achieved the best accuracy of 92.5% and an F1 score of 90%. Our comparative analysis highlights the effectiveness of the lightweight, cost-effective image recognition-based method for real-time, resource-constrained applications.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Pakistan Sign Language Recognition: From Videos to Images

  • 期刊论文

  • 发表日期：

    2025年6月2日

查看全文
  • Hafiz Muhammad Hamza,
  • Aamir Wali

摘要

Pakistan Sign Language (PSL) is the primary mode of communication for the Deaf community in Pakistan, yet research in PSL recognition remains underdeveloped. The primary challenge lies in the limitations of its dataset, the PSL Dictionary, which contains only two samples per sign, making it difficult to develop accurate recognition systems. While the action recognition models achieve high accuracy, they are resource-intensive, requiring substantial storage and computational power. To address these challenges, we propose a novel modality transformation approach that converts video-based sign language gestures into single images, allowing the use of image recognition models instead. In our approach, 21 key landmarks are first extracted from each video using MediaPipe. The movement of each landmark, such as the index fingertip, is tracked across all frames and represented as a single image per class, resulting in 21 distinct feature representations per video sample. To further enhance recognition performance, we introduce feature aggregation as part of this transformation. During feature aggregation multiple landmark images of the same class were combined into a single denser image. This integrated pipeline of modality transformation and feature aggregation allows the model to learn richer spatial representations from limited data. This reduces both storage and computational requirements while maintaining comparable performance to the action recognition models. We utilized Convolutional Neural Networks (CNN) to develop an efficient image recognition-based PSL recognition system. Although CNNs require a lot of training data, our model achieved superior performance in PSL recognition compared to existing techniques. Finally, building on our modality transformation approach, we also demonstrated the effectiveness of feature-specific ensemble learning, which achieved the best accuracy of 92.5% and an F1 score of 90%. Our comparative analysis highlights the effectiveness of the lightweight, cost-effective image recognition-based method for real-time, resource-constrained applications.