Providing sufficient and sustainable energy for users has always been a concern for all related stakeholders. The development of Deep Learning (DL) models with various applications has increased the need for their training, consequently raising their energy consumption drastically. This work has concentrated on reducing energy consumption of training the models with Fully Connected (FC) layers. So, the energy consumption of Multilayer Perceptron (MLP) models with FC layers has been investigated first to come up with a proper solution. The idea is based on the concept that the energy consumption of DL models exponentially rises by increasing their size. So, a small model with much lower energy consumption has been used for good initialization of the large model. As a result, there would be fewer number of epochs and less energy needed to reach a specific accuracy in comparison to randomly initialization of the model. The transferring of weights from the small model to the large model has been carried out by proposing a method called Proportional Uniform Distribution (PUD). The proposed method has been examined in various cases. The results show that training the models using the proposed method has lower energy consumption. Moreover, it can be applied easily to any library. Keras library has been selected in this work as a case.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Small-to-Large: Proportional Uniform Distribution Weights Transferring to Reduce Energy Consumption of Training the Models with Fully Connected Layers

  • Farzad Vazinram

摘要

Providing sufficient and sustainable energy for users has always been a concern for all related stakeholders. The development of Deep Learning (DL) models with various applications has increased the need for their training, consequently raising their energy consumption drastically. This work has concentrated on reducing energy consumption of training the models with Fully Connected (FC) layers. So, the energy consumption of Multilayer Perceptron (MLP) models with FC layers has been investigated first to come up with a proper solution. The idea is based on the concept that the energy consumption of DL models exponentially rises by increasing their size. So, a small model with much lower energy consumption has been used for good initialization of the large model. As a result, there would be fewer number of epochs and less energy needed to reach a specific accuracy in comparison to randomly initialization of the model. The transferring of weights from the small model to the large model has been carried out by proposing a method called Proportional Uniform Distribution (PUD). The proposed method has been examined in various cases. The results show that training the models using the proposed method has lower energy consumption. Moreover, it can be applied easily to any library. Keras library has been selected in this work as a case.