Multimodal fusion seeks to enhance the performance of models for various applications by extracting and integrating information from multiple modalities, such as text, images, and others. Recent studies have demonstrated the advantages of Transformer-based approaches for multimodal fusion in numerous multimedia tasks. However, Transformers often face efficiency challenges when handling long-range sequence modeling. In this paper, we address common multimodal classification tasks in social media, specifically sarcasm detection and sentiment analysis. We introduce HF-Mamba (Hierarchical Fusion based on Mamba), a novel framework designed to achieve superior multimodal fusion. HF-Mamba processes textual and visual data as sequences and leverages Mamba’s capability for long-range sequence learning to integrate information across modalities at multiple levels effectively. Experiments conducted on two widely used public datasets from Twitter and Yelp validate the effectiveness of HF-Mamba. The results demonstrate that HF-Mamba can achieve state-of-the-art performance for sarcasm detection and sentiment analysis, outperforming existing baseline models.