Optimization algorithms are essential for the effectiveness of training and performance of deep learning models, but their comparative efficiency across various architectures and data sets remains insufficiently quantified. In this study, the output of three widely used optimizers-adam, RMSPROP and lamb-system is evaluated on a convolution (Resnet-50, VGG-19), repetitive (LSTM, GRU) and reviews based on transformer across image classification (MNIST, CIFAR-10, OCT-2017) and textual (IMDB). During more than 1200 controlled experiments with extensive hyperparameter tuning, Adam achieves optimal low -data accuracy (92.3% per MNist) and transformation applications (4.2% higher BLEU scores), while RMSProp excels in convolution networks. comparison with medical display) compared to Adam) compared to Adam. Lamb proves to be a better distributed training, allowing 98% of large batch doses to be used and reduces the Imagenet era by 53%. The introduction of a normalized metric of convergence efficiency shows that adaptive methods bring 22–37% faster than SGD, but show sensitivity to learning speed plans. Critical compromises are identified, including Adam’s vulnerability to gradient sparsite in the tasks of NLP, RMSPROP robustness towards class and Lamb scalability of costly balance. These findings provide instructions that can be available, emphasizing that the optimizer’s efficiency is dependent on context and must be aligned with model architecture, data distribution and computing sources.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Exploring Optimization Algorithms in Deep Learning: A Quantitative Analysis

  • Paras Mahajan,
  • Harishchander Anandaram,
  • K. S. Shreenidhi,
  • Lakshay Madaan,
  • Aryaa Dhole,
  • Navjot Singh Talwandi

摘要

Optimization algorithms are essential for the effectiveness of training and performance of deep learning models, but their comparative efficiency across various architectures and data sets remains insufficiently quantified. In this study, the output of three widely used optimizers-adam, RMSPROP and lamb-system is evaluated on a convolution (Resnet-50, VGG-19), repetitive (LSTM, GRU) and reviews based on transformer across image classification (MNIST, CIFAR-10, OCT-2017) and textual (IMDB). During more than 1200 controlled experiments with extensive hyperparameter tuning, Adam achieves optimal low -data accuracy (92.3% per MNist) and transformation applications (4.2% higher BLEU scores), while RMSProp excels in convolution networks. comparison with medical display) compared to Adam) compared to Adam. Lamb proves to be a better distributed training, allowing 98% of large batch doses to be used and reduces the Imagenet era by 53%. The introduction of a normalized metric of convergence efficiency shows that adaptive methods bring 22–37% faster than SGD, but show sensitivity to learning speed plans. Critical compromises are identified, including Adam’s vulnerability to gradient sparsite in the tasks of NLP, RMSPROP robustness towards class and Lamb scalability of costly balance. These findings provide instructions that can be available, emphasizing that the optimizer’s efficiency is dependent on context and must be aligned with model architecture, data distribution and computing sources.