Integrating pseudo labeling with contrastive clustering for transformer-based semi-supervised action recognition

Video action recognition with semi-supervised learning is a challenging research topic due to the low-labeling ratio. Previous works mainly tackle the problem with two kinds of approaches: pseudo labeling and contrastive learning. Different from existing approaches that often treat the two parts separately, we propose an integrated learning framework that incorporates pseudo labeling and contrastive clustering in a coherent and mutually beneficial way. On one hand, the contrastive learning aggregates data from the same class into clusters, yielding more reliable pseudo labels for training the classifier; on the other hand, the re-trained classifier predicts categories for unlabeled data, thereby guiding contrastive learning to establish discriminative representations. We theoretically prove that the two iterative operations can be formulated as an E-M algorithm and validate its generalization ability upon the semi-supervised classification task with experiments. Specifically, we construct a MoCo-like structure to implement the proposed learning framework and explore the potential of employing the video tramsformer for semi-supervised action recognition. Furthermore, We also devise a global-local view sampling strategy for video data augmentation, which verifies to facilitate the representation learning and advance the performance. We implement extensive experiments on three video action recognition datasets with a series of data labeling ratios. Compared with state-of-the-art (SOTA) methods, the proposed approach achieves superior or competitive performances. For example, with \(1\%\) $1\%$ labeling ratio, the top-1 accuracy increase to \(49.1\%\) $49.1\%$ and \(52.4\%\) $52.4\%$ on UCF-101 and Kinetics-400 datasets, respectively, surpassing SOTA by \(2.8\%\) $2.8\%$ and \(3.3\%\) $3.3\%$ .