Multimodal sentiment analysis aims to utilize the combined information from different modalities to gain a comprehensive understanding of human sentiment expressions. Previous research mainly focused on the separate analysis of either global information or fine-grained information from modalities, neglecting the relationship between fine-grained features and high-level semantic representations. In this paper, we propose a framework named INFIG. This framework leverages global alignment module to extract profound emotional information and harnesses the hierarchical fine-grained alignment module to fuse fine-grained information, thereby strengthening the connectivity between image-text tokens. Furthermore, we explore the optimal combination of these two modules to maximize the model’s capability in capturing emotional expressions. Comprehensive evaluations conducted on four widely-used multimodal datasets highlight the advantages and efficacy of the approach we propose. The visualization further confirms the success of our model in integrating both fine-grained and global information, leading to better interpretability.