Hadoop Distributed File System (HDFS) is a distributed file system, which can store and manage a massive number of files. However, its performance degrades when processing huge amount of small files. In response to this problem, we propose a new strategy CHAC (Correlation and Hierarchical Ascending Clustering) for the small files problem. Our strategy aims to merge small files into large ones based on the correlation between files. In this regard, CHAC takes into account several criteria such as file size, requests number and requesting clients. The hierarchical ascending clustering is used as a grouping tool of correlated small files. Extensive carried out experiments show that the proposed strategy reduces the NameNode memory consumption. CHAC also offers a shrewd use of the DataNodes storage space through a reduced number of obtained clusters of small files by increasing the average disk utilization of data blocks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Combining Similarity-Based Correlation and Hierarchical Ascending Clustering for Small Files Problem in HDFS

  • Hanène Chettaoui,
  • Farah Hkiri

摘要

Hadoop Distributed File System (HDFS) is a distributed file system, which can store and manage a massive number of files. However, its performance degrades when processing huge amount of small files. In response to this problem, we propose a new strategy CHAC (Correlation and Hierarchical Ascending Clustering) for the small files problem. Our strategy aims to merge small files into large ones based on the correlation between files. In this regard, CHAC takes into account several criteria such as file size, requests number and requesting clients. The hierarchical ascending clustering is used as a grouping tool of correlated small files. Extensive carried out experiments show that the proposed strategy reduces the NameNode memory consumption. CHAC also offers a shrewd use of the DataNodes storage space through a reduced number of obtained clusters of small files by increasing the average disk utilization of data blocks.