While the main mechanism of supervised learning is to minimize an error cost function toward fitting the training data to the target values or labels, an overdoing of the fitting may result in a phenomenon called overfitting. In other words, an over-trained model might not provide accurate predictions for unseen data. This is particularly true when only a small number of training samples is given where only a partial picture of the unknown data distribution can be inferred. In this chapter, we focus on learning with coefficient shrinkage, which is also known as penalized learning or regularization.