Multimodal sentiment analysis (MSA) has attracted extensive attention with the rapid development of social media. How to effectively fuse heterogeneous modal features remains a core challenge to improve the performance of MSA. However, most existing MSA models fail to distinguish the unique sentiment information of visual and audio modalities as auxiliary modalities in the cross-modal fusion method with text as the dominant modality, and uniformly use the same fusion strategy. To address these problems, we propose a novel method, namely a Modality-Specific Interaction Network based on Modality Semantic Alignment. First, the Modal Alignment Prompt module is used to learn the shared semantic representations between the modalities to promote subsequent interactions. Then, the Dual Attention Fusion module is combined to mine the synergistic relationship between text and audio modalities, and the Dual Semantic Fusion module is used to strengthen the interaction between text and visual modalities. The effectiveness of the proposed method is demonstrated on the CMU-MOSI and CMU-MOSEI datasets.