Multi Modal Aware Transformer Network for Effective Daily Life Human Action Recognition

Multimodal approaches for human action recognition leverage the complementary nature of different data modalities to improve system performance in healthcare, sports, smart homes, and surveillance. However, effectively fusing features from different modalities is challenging. Generally, action recognition involves extracting spatial and temporal information from video footage. To this end, 3D poses are fed into an STGCN to effectively capture spatial relationships between body parts and the temporal dependencies between consecutive frames. Self-attention was applied in the spatial dimension to focus on joint relationships dynamically in the same frame. External attention was used in the temporal dimension to capture relevant temporal dynamics between different joints over time. A 3D CNN was applied to extract spatial and temporal information from RGB data. Both were combined using the Intermodal Attention-based Fusion (IAF) component, which makes it possible to take into account the context information modalities by enabling the model to concentrate on the highly informative aspects of each modality during fusion. By learning attention weights, the model can selectively weight the contributions of different modalities based on their relevance to the action being recognized. Tested on the NTU-RGB+D dataset, the architecture outperformed existing approaches, showing promising effectiveness.