Differential architecture search (DARTS) has emerged as a prominent research area, yet it grapples with a longstanding challenge: the discretization discrepancy problem. This issue directly impedes the search for an optimal model architecture and undermines search algorithm performance. To alleviate this issue, we propose a novel coarse gradient weighting algorithm. Our proposed algorithm has the capability to simulate the discretization process, wherein the architectural parameters move toward both ends. And we integrate this discretization process into the training phase of the architectural parameters, enabling the model to adapt to the discretization process in a trial-and-error fashion. Specifically, based on the architectural parameters in training, we divide the candidate operations into two regions, i.e., the easy-to-select region and the hard-to-be-selected region. The different weighting strategies are implemented in different regions so that the architectural parameters are pushed to the ends. The processed architecture parameters are used for training, which is equivalent to introducing the discretization process into the search phase. Additionally, we use the coarse gradient algorithm to optimize the updating process of the weighting algorithm and theoretically justify the rationality of the coarse gradient weighting algorithm. Extensive experimental results demonstrate that our proposed method can improve the performance of the searched model and make DARTS more robust without adding additional search time.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DARTS-CGW: Research on Differentiable Neural Architecture Search Algorithm Based on Coarse Gradient Weighting

  • Wenbo Liu,
  • Tao Deng,
  • Rui An,
  • Fei Yan

摘要

Differential architecture search (DARTS) has emerged as a prominent research area, yet it grapples with a longstanding challenge: the discretization discrepancy problem. This issue directly impedes the search for an optimal model architecture and undermines search algorithm performance. To alleviate this issue, we propose a novel coarse gradient weighting algorithm. Our proposed algorithm has the capability to simulate the discretization process, wherein the architectural parameters move toward both ends. And we integrate this discretization process into the training phase of the architectural parameters, enabling the model to adapt to the discretization process in a trial-and-error fashion. Specifically, based on the architectural parameters in training, we divide the candidate operations into two regions, i.e., the easy-to-select region and the hard-to-be-selected region. The different weighting strategies are implemented in different regions so that the architectural parameters are pushed to the ends. The processed architecture parameters are used for training, which is equivalent to introducing the discretization process into the search phase. Additionally, we use the coarse gradient algorithm to optimize the updating process of the weighting algorithm and theoretically justify the rationality of the coarse gradient weighting algorithm. Extensive experimental results demonstrate that our proposed method can improve the performance of the searched model and make DARTS more robust without adding additional search time.