<p>In deep learning, the choice of optimizer plays an important role in the training efficiency and final performance of models. Adam is widely used because of its good convergence and stability. However, Adam may exhibit oscillations and fluctuations in the training when the gradient varies significantly, because its second-moment uses the square of the gradient. In addition, Adam suffers from slow convergence due to its sensitivity to sparse gradients changes and overly conservative learning rates. To address these issues, we propose an improved optimizer called AdamMAD (Mean Absolute Deviation). AdamMAD performs the second-moment by using the mean absolute deviation (MAD) instead of the traditional square of the gradient, resulting in smoother Training Loss, fewer spikes and valleys, and more friendly sparse gradients processing. This paper details the design principles and specific implementation of AdamMAD, derives its mathematical formulas, and validates its performance on various datasets and models through extensive experiments. Experimental results demonstrate that AdamMAD offers better stability and convergence in the training, compared to optimizers such as Adam, AdamW, and QHAdam that update the learning rate using the second-moment.</p>