Multimodal humor recognition has gradually drawn attention in recent years, with memes serving as a prominent form of multimodal communication on the internet. However, current research in multimodal humor recognition lags behind due to the lack of publicly available datasets. To address this gap, we create the HuME meme dataset, consisting of 10,600 images paired with Chinese text, aimed at discerning whether memes are humorous. Furthermore, given humour’s metaphorical nature, existing multimodal models struggle to fully adapt to humor recognition tasks. Hence, we delve into humor theory, analyzing the textual and visual coherence within memes from syntactic and semantic perspectives. Finally, considering the high-dimensional nature of the data, we incorporate manifold learning to further represent the features of high-dimensional data. We quantitatively and qualitatively analyze the experimental results to verify the legitimacy of the HuME dataset and the effectiveness of Dual-granularity Hierarchical Fusion Network. The dataset is available at https://github.com/DericWmy/HuME .