Multimodal action recognition is a critical task for understanding complex human behaviors across different data sources. In this paper, we focus on Track 3 of the Multimodal Visual Pattern Recognition Challenge, which emphasizes recognizing human actions or activities from multimodal data streams. However, traditional methods have not sufficiently considered the complementarity of different modalities, thereby failing to fully leverage the valuable information from each. In this paper, we propose a Modality Fusion Vision Transformer (MF-ViT) network, in which we design a Modality Fusion Adaptor module that significantly enhances the model’s ability to capture complex cross-modal relationships and fine-grained features by strengthening inter-modality interactions. Our approach achieved 1st place in the MMVPR Challenge Track 3, and the experimental results demonstrate its effectiveness, achieving 100% Top-1 accuracy on the RGB-T modality, 99% Top-1 accuracy on the RGB-D modality, and 97.4% Top-1 accuracy on the RTD modality. These results demonstrate the effectiveness and superiority of our approach in the multimodal action recognition task. The code of our method is available at: https://github.com/caicai211/MF-ViT .