This paper presents a novel approach to address the issue of identity protection in facial image datasets. Our goal is to prevent any violation of privacy for the individuals depicted in the dataset while ensuring that the dataset is still useful for downstream training tasks and maintains the quality of the images. Previous methods have predominantly used Generative Adversarial Network (GAN)-based models to anonymize facial datasets. However, such models suffer from distortion and loss of detail when processing real-life images. To overcome these challenges, we propose a method for anonymizing faces using a diffusion model. Our approach retains the original facial attributes and produces anonymized images with high-quality image details. These attributes are essential when using anonymized images for downstream tasks. Our proposed framework optimizes the latent space vectors of the conditional Denoising diffusion probabilistic model (DDPM) to maintain a certain distance between the identity features (in the feature space of ArcFace) of the anonymized images and the original identity features while preserving the facial attributes. Through qualitative and quantitative experiments, we demonstrate that our method can effectively anonymize the identity of facial images while preserving image details and facial attributes.