The swift advancement of large-scale and powerful text-to-image diffusion models has sparked increasing concerns about their potential misuse in generating images of specific individuals, leading to malicious deepfake production and violations of privacy. In this paper, we present TFCM, a framework for Facial Concept Erasure that operates without tuning, designed to stop models from producing images with undesired facial identities, even in response to prompts. Current concept erasure techniques usually have a limited scope and often necessitate fine-tuning the model for each concept, making it challenging to achieve a balance between erasing specific concepts and maintaining unrelated ones. In contrast, TFCM requires no fine-tuning (i.e., it does not disrupt the model’s internal knowledge) to erase any unwanted face ID, making it more efficient and generalizable compared to previous methods. Moreover, it offers superior preservation of unrelated concepts. This is achieved by leveraging steer-face-based attention modulation and face-mask-guided sample modulation. The former introduces an additional image prompt and modulates the cross-attention level feature to steer the generation process away from unwanted face features, effectively distancing the generated image from unwanted face IDs. The latter directs the diffusion model to focus more on the regions corresponding to faces during the sampling process, thereby minimizing the impact on other areas. Extensive experiments demonstrate the effectiveness of TFCM with better performance in term of integrity, quality and generalization compared with baseline models.