Distributed Evostream Algorithm Based on Apache Spark

Clustering is pivotal in data mining, aiming to uncover hidden structures within datasets. It is extensively applied in fields like marketing and healthcare for grouping similar entities. With the advent of large-scale, rapid data sources like sensors, clustering these vast and dynamic data streams has become increasingly significant. The evoStream algorithm stands out for its capability in data stream clustering. This study delves into crafting a distributed variant of the evoStream algorithm in Python, explicitly focusing on distributing its offline phase—an evolutionary optimization technique utilized across various domains. To facilitate this distribution, the research integrates the principles of the “Master–Slave Model,” employing a master–slave architecture, and the “Island Model,” which segments the process into multiple independent units or “islands.” Each Island executes a version of the genetic algorithm tailored with unique parameters and populations. The primary aim is to evaluate the performance of the distributed evoStream algorithm, as designed using the Island Model, against the traditional, single-machine sequential evoStream algorithm. This evaluation is based on metrics such as execution time and algorithm speedup. The dissertation meticulously examines the outcomes of this comparison, shedding light on the strengths and weaknesses of both methodologies. It offers valuable insights for scholars and practitioners working on optimization with evoStream.