Distributed Computing in Training Machine Learning Models

Artificial Intelligence (AI) has been becoming one of the fastest-growing technology fields today, with wide-ranging applications in various domains. However, the demand for high-performance hardware resources is becoming a barrier to entry for the AI development market. Also, minimizing model training time to deploy systems sooner is a challenge when dealing with large datasets. To address these issues, distributed computing methods have been proposed to perform model training, of which the two most common approaches are data parallelism and model parallelism. In this study, we proposed a novel data parallelism method for training machine learning models, accompanied by the design of a novel communication process between worker machines using Python sockets. Experimentations have shown that our distributed computing method significantly improves results compared to those of centralized computing, with up to 50% saving of model training time in optimal scenarios.