Application of an improved vision transformer and optical flow fusion algorithm for human action recognition in human–computer interaction systems

Human action recognition (HAR) is essential for improving the intuitiveness and responsiveness of human–computer interaction (HCI) systems by accurately identifying actions such as walking, sitting, waving, and gesturing. Traditional two-stream architectures face challenges in capturing fine-grained temporal dynamics, handling variations in lighting and viewpoints, and maintaining computational efficiency. To overcome these limitations, the proposed Dynamic Komodo Mlipir-improved vision transformer-optical flow fusion (DKM-IViT-OFF) framework integrates an improved vision transformer (IViT) for spatial-contextual feature extraction with an optical flow fusion (OFF) module for temporal motion cues, guided by the Dynamic Komodo Mlipir algorithm (DKMA) for adaptive feature selection. RGB video sequences of human actions are collected under controlled indoor conditions, with redundant frames removed, normalization applied, and motion dynamics encoded across consecutive frames. Feature extraction is performed using CNN-based motion encoding and gradient-based temporal differencing to capture spatial-textural and motion-aware representations. IViT extracts high-level spatial and contextual features, while OFF computes pixel-wise spatiotemporal gradients. DKMA adaptively selects discriminative features to enhance accuracy and reduce computational cost. The fused dual representations use an attention-based weighting mechanism to strengthen temporal coherence and classification precision. The framework is trained end-to-end with a hybrid loss function balancing spatial and temporal learning. Experimental results on the Kaggle Human Action Recognition dataset (2,824 video clips) show superior performance, achieving 97.26% accuracy, 98.01% precision, 97.73% recall, and 97.21% F1-score, outperforming conventional Transformer-based HAR models. Implemented using the Python tool with TensorFlow, Keras, and OpenCV, the DKM-IViT-OFF framework ensures high-precision, real-time HAR suitable for gesture-based interfaces, virtual reality, and assistive systems.