Diffusion Transformer-Based Lighting Control and Identity-Preserving Personalization for Single-Person Portraits
摘要
A single-person portrait serves as a vital medium for personal expression and identity representation. With the rapid advancement of diffusion-based image generators, there has been growing interest in portrait relighting and personalized generation based on powerful text-to-image models. However, many existing approaches face several challenges, including difficulties in acquiring high-quality training data, a lack of publicly available portrait lighting datasets, high computational costs, and reliance on complex control networks that heavily modify the base models, thereby hindering scalability. Therefore, we propose PortraitLighting, a framework based on diffusion transformer (DiT), which is trained on easily accessible synthetic data. It consists of two key components: a text-conditioned triplet image generation module (LoRA) and an image-conditioned portrait relighting model (Relight). We design a novel “triplet image-composite prompt” training paradigm, where the low rank adaptation is leveraged to activate the DiT’s multi-image contextual generation capabilities, enabling low-cost expansion of the training dataset based on prompts. The Relight model jointly processed noise, image, and text condition tokens in the latent space, which minimally modifies the base model, implementing a lightweight image-conditioned control mechanism. Furthermore, our framework unifies both spatially aligned and non-aligned tasks, supporting background-based portrait relighting (spatially aligned) and identity-preserving personalization (non-aligned). Experiments demonstrate that our method achieves comparable or even superior performance compared to state-of-the-art approaches.