Deep Learning-Enhanced Hand Gesture Recognition for Robust Mobile Robot Control

Hand gesture control provides an intuitive human-machine interface for enhanced mobile robot usability, but maintaining high accuracy and real-time responsiveness in noisy, complex environments remains challenging. This research introduces a novel, synergistic deep learning framework that combines the state-of-the-art YOLOv8s object detection model with MediaPipe Hands for precise and efficient hand gesture recognition. The core innovation lies in using MediaPipe Hands as a robust preprocessing step to accurately detect and isolate the hand region, thereby minimizing background interference before classification by the YOLOv8s model. The system was trained on a custom dataset of 1,400 labeled images across five distinct gestures and tested in a real-time control pipeline, transmitting commands via Bluetooth to a mobile robot. Experimental evaluations demonstrate that this integrated approach significantly outperforms standalone YOLOv8s, achieving an accuracy exceeding 89%. Critically, under challenging high-intensity lighting and cluttered background conditions, the combined model demonstrated superior robustness, achieving an average inference accuracy of 91%. This represents a substantial improvement of over 50% compared to the standalone YOLOv8s, which significantly struggled in these environments (dropping to ~38% accuracy). This study validates the effectiveness of the combined deep neural network approach in enabling robust, reliable, and swift gesture-based robot control, advancing the field of natural human-machine interfaces.