The concept of neural network has been around long before the recent dominance of deep learning methods. This chapter introduces the early development of neural network in the 1960s to 1980s period, including the single-layer perceptron for linear separation and the multi-layer perceptron and Madaline for nonlinear separation. We will discuss the inference and training algorithm of these early neural network models, and show how they inspire the modern-day neural network design and training method.