Cloud Versus Local: Performance Evaluation of Multi-node Hadoop Clusters Using HiBench Benchmarks

This study aims to evaluate the performance difference between Hadoop clusters deployed on the Google Cloud Platform (GCP) and local machines having similar configurations. This project utilizes the HiBench benchmarking suite, focusing on critical performance metrics such as CPU Utilization, memory utilization and disk I/O performance. Both environments have identical cluster configurations, enabling a direct performance comparison. This analysis focuses on these metrics and considers factors like cost, scalability and ease of management to provide an overall evaluation. Results from the analysis explain performance advantages in GCP where CPU utilization is 15% more efficient and lower idle time, indicating better resource management. Disk I/O performance is also better on GCP where around 20% increase in throughput and shows the faster data handling capabilities. Additionally, memory utilization on GCP is optimized by nearly 10%, with lower usage percentages and more effective resource allocation. Overall, GCP’s cloud-based infrastructure substantially outperforms the on-premise setup in handling I/O-intensive workloads, offering better scalability and ease of management, which are crucial for cost-effective big data processing. This comparative analysis underscores the substantial benefits of cloud environments like GCP for deploying Hadoop clusters, particularly for organizations looking to optimize their big data infrastructure. The insights from this study will help organizations make informed decisions about selecting the right infrastructure for their data processing needs and balancing performance, cost, and operational efficiency.