Madaline uses a step function (i.e., signum) as activation, which is not smooth nor differentiable and prevents a gradient-based model update on Madaline. This problem was noted by researchers at that time; however, the limited computing architecture available in the 1960s did not support the computation of smooth activation functions in the model. It took an additional 15-20 years before researchers replaced the signum function with a smooth sigmoid function, paving the path for end-to-end gradient-based training. In this chapter, we will introduce the designs of different activation functions and the end-to-end training loop of the neural network.