Efficient Load Balancing and Data Placement Algorithm in Hadoop Distributed File Systems
摘要
Map reduce has turned into an imperative distribution processing model on behalf of scale data rigorous applications such as data mining with web indexing. The huge growth of data being generated nowadays and today is creating storage and computing is a tremendous task. In order to deal with Big Data, Distributed Computing is one of the solutions. The Big Data requires services which are provided by Distributed Computing. So many tools are available to provide the required service for big data. Hadoop is a highly demandable service for handling huge amounts of data and processing it on hardware. The distributed capacity of Hadoop gives one of the most efficient solutions for huge amounts of data. Hadoop HDFS is widely for distributed storage and Map Reduce is popular for efficient processing. Hadoop gives default data placement technique it places the data block by using random policies across the different nodes without allowing for the various execution parameters resulting in lagging of increased execution time, latency time of query processing, etc. In this, there is no control over how and where to store data blocks which leads to poor load distribution for clusters and also reduces the performance of the Map Reduce technique. Hence in this paper, we target file system management and process management to achieve better performance for big data processing. The proposed algorithm reduces the processing time by efficient load balancing and disk placement in a heterogeneous environment. It optimizes the execution time, reduces latency, increases data locality, and decreases job completion time.