With the recent advancements in diffusion-based text-to-image (T2I) models, generating high-quality, human-centric images has become increasingly easy. However, T2I models cannot preserve the fine-grained skin color information from reference images, raising potential AI ethics concerns. In this work, we propose Skin-Adapter, the first work on preserving fine-grained skin color information in text-to-image generation. To achieve this goal, we first devise the frequency-based adaptive color histogram to accurately represent the user’s skin color information. Additionally, we introduce the color distribution matching reward to explicitly enhance skin color consistency between input and output images. Experimental results show that our Skin-Adapter can maintain the fine-grained skin color information of input images in the generated images. Furthermore, we validate the superiority of our approach through quantitative and qualitative comparisons against possible alternatives.