Evaluating Support Vector Machines with Multiple Kernels by Random Search

The minimal complexity support vector machine with a three-kernel structure (ML1 SVM (MK)) has a reduced number of hyperparameters for multiple kernels and is optimized in three-stage cross-validation: In the first stage, the hyperparameter values for the L1 SVM (MK) are determined fixing the hyperparameter values for multiple kernels. In the second stage the hyperparameter values for the multiple kernels are determined, and in the third stage, the value of the hyperparameter of the ML1 SVM (MK) to control the maximum margin is determined. In this paper, we examine whether this training strategy and the reduction of hyperparameters do not deteriorate generalization performance due to falling in the local minimum. To do this we compare the results with cross-validation by random search of ML1 SVM (MK) with the full three-kernel structure. By computer experiments, we find that the generalization abilities by the three-stage cross-validation for the reduced kernel structure is statistically comparable to or better than random search for the two-class problems and most of the multiclass problems tested.