Comprehensive analysis of abnormal changes in anatomical structures in two-dimensional grey-scale ultrasound images and blood flow change characteristics in color Doppler images can be more conducive to the identification of ventricular septal defect diseases (VSD). Starting from the perspective of multi-modality, this paper designs a multi-modality correlation learning network (MC-Net) for VSD identification. MC-Net performs correlation analysis on multi-modality features from two perspectives: network structure and the image itself. In terms of network structure, this paper first constructs dual-branch feature cross-fusion blocks (CFB) to encode the associated information between different modalities to achieve the fusion of global features and local features and then performs reinforcement learning on the fused features through a series of hybrid learning blocks (HLB). In terms of the image itself, this paper designs a group selection transformer (GST) to capture the correlation between image tokens and their context, prompting the network to focus on the region of interest more effectively. This paper conducts experimental analysis in multi-modality five-chamber and parasternal short-axis views. The experimental results show that the identification performance of the proposed algorithm is better than that of the comparison methods.