Currently, mainstream unsupervised hashing retrieval methods construct an affinity matrix by mining semantic correlations among data to guide the learning of hashing functions. However, these methods overlook the multidimensionality of information, leading to the constructed similarity matrix suffering from insufficient semantic associations. Additionally, existing deep hashing methods adopt an end-to-end approach to uniformly learn high-dimensional feature extraction networks and low-dimensional hashing mapping networks, yet they fail to consider the imbalance in learning capabilities between high-dimensional and low-dimensional spaces. To tackle these issues, we propose the Unsupervised Multi-level Semantic Fusion-Driven Hashing for Cross-Modal Retrieval (MSFH) method. This method employs multi-level feature decomposition and fusion to mine semantic associations across different information dimensions. Simultaneously, it corrects neighborhood deviations between the affinity matrix and aggregated features during the learning process, constructing a robust and semantically rich affinity matrix. Finally, to balance the network imbalance between the feature extractor and the lightweight hashing function, we utilize Quality-Filtered Contrastive Learning to alleviate the information burden on the hashing function during its learning process. Experiments demonstrate that the MSFH method achieves significant performance improvements across multiple datasets. Codes can be available at https://github.com/dh2425/MSFH .