Big Data Algorithms, MapReduce and Hadoop ecosystem

This chapter discusses Big Data algorithms that are capable of processing large volumes of data by using either parallelization or streaming mode. We will look at the MapReduce algorithm for parallel processing of large amounts of data that provides a basis for many other algorithms and applications working with Big Data. The MapReduce programming model and its implementation in Hadoop were invented to address limitations of the traditional high-performance computing programming model such as MPI (Message Passing Interface) for processing very large datasets such as of web scale that actually cannot be processed on one even very big computer (“MapReduce Tutorial, Apache Hadoop, Version 3.3.6, 13 June 2023,” [Online]. Available: https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html ). MapReduce is specifically oriented on using commodity computer clusters. Hadoop is an Open Source implementation of MapReduce. It is the main computing platform for web data processing by web search engines such as Google, Bing, Yahoo!, LinkedIn, and others. We will look at the Apache Hadoop platform and ecosystem that provides a generic implementation of MapReduce and includes a wide rage of applications, libraries and software packages to store, process and visualize Big Data. Hadoop became a standard-de-facto and ultimate platform for building highly scalable Big Data applications. The Hadoop platform is provided by all big cloud providers. The chapter also provides overview of Hadoop Distributed File System (HDFS), Apache Hive and Apache Pig which are important components of the Hadoop ecosystem specifically design for storing and processing Big Data.