Multimodal sentiment analysis(MSA) aims to recognize human emotions by integrating information from multiple modalities. Previous approaches to modality decomposition for obtaining common and private representations often overlooked the fact that the interactions between modalities are actually independent, leading to inaccurate decomposed representations. To address this issue, we propose a method called Inter-Modal Shifting and Intra- Adaptation (ISIA) for the MSA task. Firstly, inter-modal shifting sequentially identifies a primary modality from all modalities and calculates the shifting magnitude relative to the other modalities within independent feature spaces, obtaining accurate common representations for each modality. Secondly, a gated attention mechanism, combined with the common representations, adaptively extracts private representation from the original modality representation. ISIA enables a more precise decomposition of modalities, enhancing the quality of both common and private representations. Experiments on two public benchmarks demonstrate that our ISIA outperforms state-of-the-art methods, confirming the model’s effectiveness. Our code is available at https://github.com/AnleKer/ISIA .