Image fusion aims to amalgamate information from multiple source images into a single fused image, retaining all details of the source image and offering comprehensive features for advanced vision tasks. However, current methods often ignore human perception, resulting in an imbalance between intrinsic and cross-functional characteristics. To address this, we propose a novel unified image fusion framework with human-like perception-driven learning. Firstly, the Swin-Transformer-based private encoder (SwinEncoder) extracts individual features, while the Swin-Transformer-UNet shared encoder (ST-UNet) captures shared features, mimicking human observation of image specifics and similarities. Introducing a novel perception fusion block (PFB) with pixel-level constraints enables the extraction of cross-information while preserving shared features. Subsequently, the ST-based global reconstruction and CNN-based local reconstruction layers produce the fused image. Extensive experiments across infrared-visible, medical, multi-exposure, and multi-focus image fusion domains showcase the promising results and generalization of our approach.