Gesture and Speech Recognition for Real-Time Multi-modal Human–Robot Interaction Using Deep Learning Based Approach

Interactions with robots involve various modes, including gestures, audio commands, and texts, where the choice of communication interface significantly influences the robot’s efficiency. Multi-modal communication, offering multiple interaction channels, proves crucial in facilitating effective human–robot interactions. In this research, a system is proposed that employs a deep learning-based gesture recognition framework integrated with a live speech recognition library that is simulated in a software environment to depict the working of multi-modal human–robot communication. The pipeline developed includes MediaPipe for pose-based feature extraction along with a transformer backbone trained to classify five hand gestures. The proposed architecture is trained with a publicly available dataset from Kaggle with 695 videos, where each action is 30 frames in duration. The model is validated in real time along with a custom dataset consisting of unseen users. This novel, computationally less expensive approach achieved improved results when compared to the state-of-the-art by producing 96% model validation accuracy with 92.37% accuracy for unseen users. For audio recognition, OpenAI’s Whisper model is utilized to perform real-time transcription with phrase extraction to map the transcription to executable commands. An alternative approach using Llama-2 large language model is tested to map the transcriptions more effectively. The integration of both modalities is realized using OpenAI Gym, a robotic environment simulator that efficiently carries out the commands given. Achieving high accuracy enables seamless and natural multi-modal communication between a human and a robot. The experiments carried out validate the feasibility and effectiveness of having a multi-modal interaction system that is compatible with robotic environments.