Patients with valvular heart disease often exhibit motion characteristics such as artery movements and anatomic characteristics, thus extracting dynamic features from coronary angiography (CAG) is of great significance for diagnosing. Given the challenge of limited annotated medical imaging data, we propose a novel self-supervised learning framework that integrates masked video modeling (MVM) and video contrastive learning, enabling the model to learn representations with both strong instance discriminability between video segments and local perceptibility between neighboring frames. Specifically, our framework consists of three key components: an off-the-shelf frozen encoder, an online encoder-decoder following the MVM pipeline and a momentum encoder composed of an exponential moving average of previous students. We enhance the integration of contrastive learning and MVM in mainly two ways: the frozen encoder converts the supervision of masked reconstruction from low-level pixels to high-level features; an augmentation strategy called frame shifting, is introduced specifically for video contrastive learning. To validate the effectiveness of our proposed method, we first conducted self-supervised pre-training on over 50,000 self-collected, unlabeled CAG sequences. Subsequently, we performed supervised fine-tuning using two small-scale labeled CAG diagnostic datasets, achieving state-of-the-art performance (98.1% and 75.0% F1-Score, respectively) in both supervised and self-supervised video recognition domains. Our code is publicly available at: https://github.com/ZmingShao/ConMVM .