A Methodology and Tool for Automatic Workload Distribution. A Case Study on Federated Learning
摘要
This paper presents a novel methodology and tool for automatic workload distribution in federated learning environments, in order to address the challenges of security and privacy in distributed machine learning systems. The proposed approach is designed to optimize the performance of federated learning by adapting workload distribution based on the heterogeneous nature of edge devices. The methodology is implemented through a Jupyter notebook extension that facilitates the execution of federated learning tasks in a distributed computing context, leveraging Docker for containerization and an integrated skeleton based compiler for parallelization tasks and environments configuration, through decorators directly in the cells. The Jupyter Workflow kernel leveraging the streamflow library in order to execute workflow in heterogeneus environments. The paper discusses the implementation details, privacy preservation mechanisms, and performance evaluation of the proposed solution, demonstrating its effectiveness in enhancing federated learning workflows.