Deep learning is a subset of machine learning that uses deep artificial neural networks (ANNs) to perform humanlike tasks such as speech recognition, image identification, and decision-making. In this chapter, the history of deep learning, including its root from and the preceding work in shallow NNs, will be introduced in terms of three waves first. Critical insights will be gained for the answers to some common questions for deep learning: “Is deep learning just a rebranding of neural networks?” “What are the major breakthroughs that drive the development of deep learning from all the way back in NNs to where it is?” “What deep learning can do and will be heading to?” After that, two basic elements of modern deep learning that help address vanishing gradients, i.e., activation and initialization, will be introduced. Next, the implementations of two most widely used types of deep NNs, i.e., CNN and RNN, especially the backpropagation through these networks, will be described. For CNN, detailed treatments in the backpropagation for convolution, padding and stride, ReLU, and pooling will be explained with examples. For RNN, the mathematical formulation of a typical RNN architecture will be provided. Based on that, practical deep learning skills will be first shared for various widely accepted initialization and batch normalization methods. Also, gradient descent optimizers as an essential part of deep learning will be investigated with details for common solvers including SGD, Momentum, NAG, Adagrad, Adadelta, RMSProp, AdaMax, and Nadam. More information about data preprocessing and augmentation will be provided at the end.