Gesture Recognition on Video Data

Recognizing dynamic hand gestures presents a significant challenge for artificial intelligence systems, as it requires integrating visual information across a sequence of frames. Current approaches to hand gesture recognition face limitations. Some methods rely solely on visual inputs, processing them without considering occlusion and illumination issues, leading to performance degradation. Others employ wearable sensors, which can be bulky and prone to noise. Alternatively, recurrent neural network-based models have been proposed to classify gestures using transformed hand keypoints in a sequential manner. To address these challenges and identify existing problems and bottlenecks, we conduct experiments using different modalities, datasets, and existing solutions. We analyze various datasets, evaluate their quality [1, 65], and select the most suitable ones for each modality, including text and images. Ultimately, we propose an artificial intelligence system inspired by Contrastive Language-Image Pretraining, leveraging the notion that hand gestures represent their own language. Our system aims to link textual representations of gestures with their visual inputs. The system comprises two main components: a pre-trained text encoder that transforms textual labels of dataset examples into meaningful embeddings, and a vision-transformer-based image sequence encoder that generates the desired embeddings from the visual modality. We pretrain the text encoder and subsequently use it to train the image encoder with text supervision.