Noise-Robust Weighted Logistic Regression Based on Outlier Detection with Expectation Maximization
摘要
Logistic regression (LR) is one of the most commonly used classification models in Machine Learning. Even though LR was shown to be asymptotically the best linear classifier, it can typically be at the cost of high variance when the sample size is small compared to the number of predictive variables. Moreover, LR models are extremely sensitive to outliers. The influence of outliers has traditionally been handled by using weighted LR so that data points considered not to be outliers have a weight equal to 1, whilst those considered outliers are assigned a lower weight, which is typically computed from some distance measure over the training data. In this paper, we propose a novel iterative procedure for estimating the parameters of an LR model that automatically accounts for the presence of outliers by following an expectation maximization (EM) approach. During the expectation step, each training data point is weighted according to its probability of being an outlier; in the maximization step, the parameters of the LR model and the outlier probabilities are updated. We have experimentally validated our proposal using a set of well-established benchmark datasets for classification. The results of the experiments show that, in addition to being computationally tractable, our new method outperforms both plain LR and several outliers detection methods, in terms of accuracy and logarithmic loss.