Differential privacy (DP) is a widely used technique for securing the privacy of data in deep learning (DL). However, despite its broad adoption, the effectiveness of DP in safeguarding privacy has not yet been challenged. This paper introduces a novel approach, a fast and robust DP optimizer for general DL tasks, which addresses the limitations of existing methods and offers a fresh perspective on the issue. We first examine the convergence condition of current Differential Privacy (DP) algorithms. Then, we propose a new general optimization algorithm called ACNSI to reduce the effect of noise. ACNSI comprises five steps: affine transformation, clipping, noising, smoothing, and inverse affine transformation. By using affine and inverse affine transforms, we can significantly reduce the added Gaussian noise, which not only improves existing differential private stochastic gradient descent methods but also holds promise for enhancing the practical application of DP in DL tasks. We provide a theoretical analysis for privacy-preserving properties and utility improvement with noise reduction and smoothing effects. Finally, we conduct an empirical study to demonstrate that ACNSI achieves the best accuracy with a tight privacy budget and economical computational cost compared to existing methods.