Analyzing System Logs of Big Data Distributed Environment: A Review

Nowadays, data generation has presented all across different aspects of life. The objective of this paper is to investigate existing individual techniques and present a comprehensive analysis of log data within distributed big data environments. It focuses on the features of big data and the widespread utilization of the Hadoop Framework as a big data application. Based on the reviewed papers, it was observed that most researchers utilized MapReduce, a key component of the Hadoop Framework, for log file analysis, while HDFS was employed for data storage. Previous researchers have also combined the Hadoop Framework with other tools and algorithms for analytical purposes. This paper provides a thorough evaluation of Hadoop’s performance in analyzing diverse log file types and presents actionable recommendations for future work. The lack of comprehensive analysis for research efforts done on distributed system log datasets and the utilization of deep learning algorithms in various application domains are impeding the progress and advancement in this field.