The essential task of extracting useful patterns from large datasets enables multiple industries to implement data-driven decisions throughout the current significant data era. Unsupervised machine learning contains clustering as one vital technique which identifies data similarities through shared common features to produce groupings. Traditional efficient algorithms such as K-Means struggle to handle complex datasets because distance-based relationships do not describe all pertinent data associations. The proposed study implements Constrained K-Means because its algorithm utilizes must-link and cannot-link constraints to preserve domain-specific relationships in cluster groupings. We conducts tests of the proposed algorithm through Apache Spark platform due to its ability to maximize distributed processing capabilities for large-scale datasets. Experimental results demonstrate that the Constrained K-means algorithm achieves better clustering performance and interpretability and marks superior results compared to K-means when dealing with constraint-aware data sets on synthetic and real-world datasets. The studied combination of constraint-based clustering and distributed computing systems demonstrates potential to create stronger yet scalable machine learning solutions for big data environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Apache Spark Implementation of the Constrained K-Means Clustering Algorithm

  • Nguyen Quang Huy,
  • Vu Thu Diep,
  • Phan Duy Hung

摘要

The essential task of extracting useful patterns from large datasets enables multiple industries to implement data-driven decisions throughout the current significant data era. Unsupervised machine learning contains clustering as one vital technique which identifies data similarities through shared common features to produce groupings. Traditional efficient algorithms such as K-Means struggle to handle complex datasets because distance-based relationships do not describe all pertinent data associations. The proposed study implements Constrained K-Means because its algorithm utilizes must-link and cannot-link constraints to preserve domain-specific relationships in cluster groupings. We conducts tests of the proposed algorithm through Apache Spark platform due to its ability to maximize distributed processing capabilities for large-scale datasets. Experimental results demonstrate that the Constrained K-means algorithm achieves better clustering performance and interpretability and marks superior results compared to K-means when dealing with constraint-aware data sets on synthetic and real-world datasets. The studied combination of constraint-based clustering and distributed computing systems demonstrates potential to create stronger yet scalable machine learning solutions for big data environments.