Combining Similarity-Based Correlation and Hierarchical Ascending Clustering for Small Files Problem in HDFS
摘要
Hadoop Distributed File System (HDFS) is a distributed file system, which can store and manage a massive number of files. However, its performance degrades when processing huge amount of small files. In response to this problem, we propose a new strategy CHAC (Correlation and Hierarchical Ascending Clustering) for the small files problem. Our strategy aims to merge small files into large ones based on the correlation between files. In this regard, CHAC takes into account several criteria such as file size, requests number and requesting clients. The hierarchical ascending clustering is used as a grouping tool of correlated small files. Extensive carried out experiments show that the proposed strategy reduces the NameNode memory consumption. CHAC also offers a shrewd use of the DataNodes storage space through a reduced number of obtained clusters of small files by increasing the average disk utilization of data blocks.