Inter-class margin climbing with cost-sensitive learning in neural network classification

Large margin stands as an intuitive indicator of reliable classifiers, reflecting classifier robustness and generalizability. However, due to the intricate nonlinear mapping, directly defining margin as the optimization objective for multi-layer neural network is always challenging. On the other hand, the cost sensitivity coefficients of individual samples hold promise for shaping decision boundaries of neural network classification. A question arises as to whether optimizing neural network classifier can be guided to achieve larger classification margin by varying instance-level cost sensitivity factor. Inspired by above question, this paper proposes a heuristic hard mining strategy designed to progressively identify challenging samples and amplify the output margin through cost-sensitive learning. The refinement process adjusts the sample distribution when optimization reaches a local minimum to ensure the sustainable optimization, ultimately leading to margin climbing. Two hard mining algorithms are designed for binary and multi-class classification problems, which utilize distinct margin definitions based on different decision-making scenarios. In the proposed method, we focus on establishing individualized margin between distinct categories to more accurately characterize the inter-class margin. Empirical results demonstrate that our proposed methodology enhances both the accuracy and robustness in neural network classification.