A Scalable Distributed Computation Framework for Tackling Underutilization and Ad-Hoc Computations in Heterogenous Clusters

Our proposed distributed computation framework addresses the issue of underutilized computing resources in institutions, companies, and communities, by providing a novel automated and efficient solution for ad-hoc distributed computing that is easy to use and utilizes several heterogenous semi-idle computing devices to provide large scale distributed parallelism, managing all setup, communication, resource allocation, and fault tolerance without any secondary input from the user. The framework will in an automated manner, find appropriate idle resources, break up the job into smaller jobs that can be balanced across several resources, manage communication and synchronization between these jobs and computational nodes, and return the result to the user. While there have been advances at tackling underutilization, they have been largely constricted to specific environments lacking genericity, complex to employ and not suitable for ad-hoc natured tasks. The framework novelly provides an extremely easy to use interface for cluster setup and job submission, ensures that the original responsibilities of the nodes are not disturbed, and further provides resumability of affected jobs, smart distributed parallelism, inter-node task balancing, and reliability for the user. The efficiency of the framework is evaluated, and it is demonstrated to achieve over 40% faster task execution and far better resource utilization across a cluster. Overall, our framework provides an effective and efficient solution for utilizing underutilized computing resources for ad-hoc distributed computing tasks.