In the process of image restoration, in order to realize the interaction between structure and texture information in the process of structural reconstruction and enhance the semantic authenticity of the restored image, on the basis of the original dual-stream generation network, we improve a dual-stream structure image recovery network based on multi-scale feature fusion algorithm. The network realizes the information interaction between structural features and texture features in space and channel in the way of feature enhancement, realizes the effective use of texture and structural information, and is conducive to generating more realistic images and preserve more semantic information. A multi-scale feature fusion network with concatenate structure is constructed to achieve the global consistency of the fusion image. By comparing experiments with other restoration networks on the Paris Street View, CelebA and Places2 datasets, it is proved that the improved image inpainting method has better objective evaluation indexes, more effective repair of the structure and texture information of damaged images, and better image inpainting performance.