Human action recognition holds significant potential for applications in various scenarios. However, existing works primarily focus on the recognition itself, failing to explore key inherent information entailed in human actions/behaviors, namely, emotions. Therefore, this work proposes DeemCLIP, a CLIP framework with Dual Emotion Enhancement Modules for multimodal video action recognition, which explicitly joint modeling action and emotion features. Specifically, DeemCLIP utilizes a pre-trained text encoder and a pre-trained visual encoder to extract multimodal features of human action videos, and introduces a textual emotion extraction module and a visual emotion attention module to extract emotional features of text and video, respectively. The contrastive learning technique is adopted to finetune the pre-trained text encoder and visual encoder. By introducing the dual emotion enhancement module, DeemCLIP enables the multimodal action recognition model to incorporate information from the external emotion context. The comparison results with competitive methods on different datasets show that the proposed DeemCLIP method reaches higher accuracies and simultaneously recognizes entailed emotion tendencies. The code is open sourced at: https://github.com/Alsacen/DeemCLIP.git