Vision-Language Pre-trained (VLP) models have shown significant ability in many video tasks. For action recognition, recent studies predominantly use meticulously designed prompt tokens or positional encodings to adapt VLP models to video domains, consequently leading to a reliance on designing and learning processes. Moreover, in mainstream fine-tuning settings, models are guided by downstream tasks, which is a coarse-grained objective toward temporal modeling. To address these issues we propose an Explicit Temporal Modeling (ETM) method that mainly consists of two key designs and is decoupled from the image model. To add temporal supervision, we focus on frame-sequential order and design a temporal-related task in a contrastive manner. To reduce dependence on the quality of design and learning when modeling temporality, we propose a module with temporality-aware computation approaches and make it compatible with the newly added task. Extensive experiments are conducted on real-world datasets, demonstrating that our proposed ETM can improve VLP models’ performance on action recognition tasks. Besides, our model also performs generalization ability in few/zero-shot tasks. Code and supplementary are available at https://github.com/lyxwest/ETM .