In deep learning, a phenomenon has recently been reported, in which the generalization performance is rapidly improved by learning a model until losses become almost zero during learning, and by continuing learning for a further large number of epochs after over-fitting. This is known as “Grokking”. In addition, “Flooding” is a method to improve recognition accuracies by learning a model to control that its loss becomes non-zero during a learning process. In this paper, we report on the effect of applying Flooding in a situation where Grokking occurs during learning a model, using a controlled environment with specific datasets, deep learning models, and hyper-parameters. In the evaluation experiments, we have confirmed that the application of Flooding sometimes improves recognition accuracies even in situations where Grokking occurs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Behavior Analysis of Grokking with Flooding

  • Hiroto Ito,
  • Minoru Mori

摘要

In deep learning, a phenomenon has recently been reported, in which the generalization performance is rapidly improved by learning a model until losses become almost zero during learning, and by continuing learning for a further large number of epochs after over-fitting. This is known as “Grokking”. In addition, “Flooding” is a method to improve recognition accuracies by learning a model to control that its loss becomes non-zero during a learning process. In this paper, we report on the effect of applying Flooding in a situation where Grokking occurs during learning a model, using a controlled environment with specific datasets, deep learning models, and hyper-parameters. In the evaluation experiments, we have confirmed that the application of Flooding sometimes improves recognition accuracies even in situations where Grokking occurs.