<p>In the field of infrared and visible image fusion, researchers have put increasingly complex fusion networks forward to pursue better fusion metrics. This has led to a growing number of parameters in fusion models. Additionally, most fusion models rarely address the issue of preserving background details in images, while these details are vital to subsequent advanced visual tasks, such as image analysis and recognition. In response to these limitations mentioned above, this paper proposes a novel image fusion algorithm called lightweight multi-scale hierarchical dense fusion network (LMHFusion). Concisely, we propose a lightweight multi-scale encoder. It extracts multi-scale features from input images through four encoding blocks with different receptive fields. Then, a designed hierarchical dense connection method is employed to concatenate distinct scale features. Unlike traditional manual fusion strategies, our fusion network is designed to be learnable and has adjustable weights. Moreover, we have specially designed a histogram equalization loss to train LMHFusion. This new loss produces fused images that contain both prominent structures and rich details. Through comparative analysis of LMHFusion and twelve other representative fusion models, it has been proven that LMHFusion can make the model more suitable for resource-constrained scenarios apart from enhancing the quality and visual effects of fused images. Our model is nearly 5000 times smaller in size compared to RFN-Nest.</p>