Decision Boundaries in Feature Space

Decision boundaries partition feature space so that samples from different classes are in different compartments. Classification applies a predicate on the features of an unknown sample that returns one of the compartments. Linear decision boundaries for two-class problems will be presented first. The predicate is the sign of a linear equation applied to features of a sample. The decision boundary is a parametrized linear equation where, ideally, the prediction for all samples from training data is true. Support vector machines extend the concept by adding a condition to search for a boundary that is furthest away from all samples. Conditions will be presented, where the distance from the decision boundary is related to the conditional a posteriori probability of a sample to belong to a class given its features. Logistic regression is then introduced as means to find an optimal decision boundary under these conditions. Finally, extensions to multiclass (multinomial) problems and to non-linear decision boundaries are presented. The chapter is concluded with a section on model ensembles. Instead of using ever more complex classifiers, ensemble models achieve a performance gain by combining simple classifiers. Two different strategies for this, bagging and boosting, are introduced.