Small-to-Large: Proportional Uniform Distribution Weights Transferring to Reduce Energy Consumption of Training the Models with Fully Connected Layers
摘要
Providing sufficient and sustainable energy for users has always been a concern for all related stakeholders. The development of Deep Learning (DL) models with various applications has increased the need for their training, consequently raising their energy consumption drastically. This work has concentrated on reducing energy consumption of training the models with Fully Connected (FC) layers. So, the energy consumption of Multilayer Perceptron (MLP) models with FC layers has been investigated first to come up with a proper solution. The idea is based on the concept that the energy consumption of DL models exponentially rises by increasing their size. So, a small model with much lower energy consumption has been used for good initialization of the large model. As a result, there would be fewer number of epochs and less energy needed to reach a specific accuracy in comparison to randomly initialization of the model. The transferring of weights from the small model to the large model has been carried out by proposing a method called Proportional Uniform Distribution (PUD). The proposed method has been examined in various cases. The results show that training the models using the proposed method has lower energy consumption. Moreover, it can be applied easily to any library. Keras library has been selected in this work as a case.