Towards exploiting linear regression for multi-class/multi-label classification: an empirical analysis

Regression and classification are the two main learning tasks in supervised learning, and both of them can be solved by learning a hyperplane from training samples. However, the hyperplane in regression task aims at approximating the labels of samples as much as possible, while the hyperplane in classification task aims at separating the samples belonging to different classes as much as possible. From this perspective, regression and classification are two completely different learning tasks. However, linear regression is often used to solve multi-class/multi-label classification problems, which can be decomposed into a set of binary classification problems. In this paper, we focus on analyzing the issues of regression models in classification tasks. Firstly, when \(\{-1, +1\}\) $\{-1,+1\}$ is used to denote negative and positive class, we derive that it is essentially equivalent to optimizing square loss as the surrogate loss function of zero-one loss to solve binary classification problem via learning linear regression model. Then, we also derive what will happen to the model when \(\{-1, +1\}\) $\{-1,+1\}$ is replaced with \(\{0, 1\}\) $\{0,1\}$ for three different versions of linear regression. Finally, extensive experiments are conducted over multi-label/multi-class classification tasks and corresponding discussions are further conducted according to the experimental results.