MakeupDiffuse: a double image-controlled diffusion model for exquisite makeup transfer

Makeup transfer is a challenging task, involving the transfer of a reference makeup style onto the source face while preserving the original appearance. Current GAN-based methods, representing makeup styles through reduced-dimensional matrices, often generate smooth high-frequency attributes and imprecise images. Additionally, these models are difficult to train and prone to model collapse problems. This paper introduces MakeupDiffuse, a diffusion-based model adapting a foundational diffusion model pre-trained on large-scale image datasets for makeup transfer. Fine-grained makeup transfer is achieved through a novel double image controller, controlling the identity process and adjusting the makeup style. To address the lack of paired data, we include another pre-trained makeup transfer network as a teacher module to supervise model training. Due to the flexible architecture, our model efficiently trains and generates faces with high perceptual quality and the expected makeup style. Unlike subjective evaluations based on user studies, we propose a new objective and comprehensive quantitative metric: Makeup Transfer Score, improving the current evaluation system. Extensive experiments demonstrate our approach’s ability to generate natural makeup faces with exquisite details, achieving state-of-the-art performance.