Scholarlink

<p>We contribute to the understanding of learning behavior of neural networks by showing that in principle a few possibly noisy samples can significantly alter weights of a network, even after most samples have been fit. Formally, we approach learning dynamics by analyzing prediction accuracy, input reconstruction ability, and prediction performance. A neural network can exhibit a prototype-learning phase decreasing reconstruction loss initially, possibly increasing the loss of a small set of well-defined samples, i.e., defined based on the L2-norm and dot product with the mean of class samples. Toward the end of training, parameter updates might mostly reduce the classification loss of a few samples, which increases reconstruction loss. Aside from providing a mathematical analysis of a linear network, we also assess the behavior using common datasets and architectures from computer vision. On the practical side, our work supports training data selection with low computational effort, i.e., identifying samples that are more likely to be misclassified. This can help in identifying labeling errors, ensuring a diverse dataset, and data valuation. Our work also casts a different view on the notion of outliers in supervised learning.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Learning in NN: from fitting most to fitting a few

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年8月27日

查看全文
  • Johannes Schneider

摘要

We contribute to the understanding of learning behavior of neural networks by showing that in principle a few possibly noisy samples can significantly alter weights of a network, even after most samples have been fit. Formally, we approach learning dynamics by analyzing prediction accuracy, input reconstruction ability, and prediction performance. A neural network can exhibit a prototype-learning phase decreasing reconstruction loss initially, possibly increasing the loss of a small set of well-defined samples, i.e., defined based on the L2-norm and dot product with the mean of class samples. Toward the end of training, parameter updates might mostly reduce the classification loss of a few samples, which increases reconstruction loss. Aside from providing a mathematical analysis of a linear network, we also assess the behavior using common datasets and architectures from computer vision. On the practical side, our work supports training data selection with low computational effort, i.e., identifying samples that are more likely to be misclassified. This can help in identifying labeling errors, ensuring a diverse dataset, and data valuation. Our work also casts a different view on the notion of outliers in supervised learning.