In recent years, text-to-image generation has witnessed rapid advancements, among which personalized facial generation has garnered significant attention. This technology has been widely applied in areas such as virtual avatar construction and creative content generation. However, existing methods exhibit poor performance when generating face images for minority or underrepresented race-gender groups. They often result in feature entanglement, where non-target race-gender attributes are fused into the output, leading to distortion, identity loss, and fairness issues. To address this challenge, we propose FairMoE, a Mixture of Experts (MoE) framework assigning decoupled experts to each race-gender category. FairMoE stores knowledge for various groups in different experts and dynamically routes to the appropriate expert according to the generation goal. This approach, by decoupling knowledge, effectively mitigates racial feature entanglement. Additionally, we develop AttrCtrl, a structured attribute controller optimized through cyclic feedback, which can provide fine-grained semantic control. Extensive experiments demonstrate that our method achieves superior attribute diversity, semantic alignment, and fairness, outperforming existing SOTA methods across multiple benchmarks. Specifically, it achieves a CLIP score of 0.2602, which is a 34.2% improvement over Textual Inversion, and a face detection score of 0.9808, exceeding DreamBooth by 22.6%.