Auxiliary text semantics can help Vision Language Models better understand video content by providing additional textual information. However, current methods over-relying on non-action entities, like subjects and scenes, weakens fine-grained action semantics. Additionally, the global feature alignment strategy also ignores action temporal dynamics, making it hard to capture complex action’s temporal semantic differences. To address these issues, this paper proposes a Transformer-based video action recognition model (TASVARM), which integrates temporal understanding of action semantics through cross-modal interaction and modality fusion. In the model, we expand an action knowledge base to create multi-fine-grained text prompts, design a Temporal-Semantic Collaborative Encoder to align text with video frames through cross-attention, and use a Semantic Guidance Module to fuse spatio-temporal features with action semantics, enhancing the model’s focus on key action regions. Four widely used datasets, including Charades, Animal Kingdom, UCF101, and HMDB51, are used to test the model. The experimental results demonstrate that our approach achieves highly competitive performance.