Scholarlink

<p>Neural-ODE parameterizes a differential equation using continuous depth neural network and solves it using numerical ODE-integrator. This model offer a constant memory cost compared to models with discrete sequence of hidden layers in which memory cost increases linearly with the number of layers. In addition to memory efficiency, other benefits of Neural-ODE include adaptability of evaluation approach to input, and flexibility to choose numerical precision or fast training. However, despite having all these benefits, it still has some limitations. We identify the ODE-integrator (also called ODE-solver) as the weakest link in the chain as it may have consistency, convergence&#xa0;and stability (CCS) issues and may suffer from slower convergence or may not converge at all. We propose a first-order Nesterov’s accelerated gradient descent(NAG) based ODE-solver which is proven to be stable, consistent and convergent. We empirically demonstrate the efficacy of our approach by training faster, while achieving better or comparable performance against Neural-ODE employing other fixed-step explicit ODE-solvers as well&#xa0;as discrete depth models such as ResNet in three different tasks including supervised classification, density estimation, and time-series modelling.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

On Tuning Neural ODE for Stability, Consistency and Faster Convergence

  • 期刊论文

  • 发表日期：

    2025年3月25日

查看全文
  • Sheikh Waqas Akhtar

摘要

Neural-ODE parameterizes a differential equation using continuous depth neural network and solves it using numerical ODE-integrator. This model offer a constant memory cost compared to models with discrete sequence of hidden layers in which memory cost increases linearly with the number of layers. In addition to memory efficiency, other benefits of Neural-ODE include adaptability of evaluation approach to input, and flexibility to choose numerical precision or fast training. However, despite having all these benefits, it still has some limitations. We identify the ODE-integrator (also called ODE-solver) as the weakest link in the chain as it may have consistency, convergence and stability (CCS) issues and may suffer from slower convergence or may not converge at all. We propose a first-order Nesterov’s accelerated gradient descent(NAG) based ODE-solver which is proven to be stable, consistent and convergent. We empirically demonstrate the efficacy of our approach by training faster, while achieving better or comparable performance against Neural-ODE employing other fixed-step explicit ODE-solvers as well as discrete depth models such as ResNet in three different tasks including supervised classification, density estimation, and time-series modelling.