In today’s digital transformation, big data plays a major role in fostering innovation and guiding strategic direction. Big data offers many opportunities; it also introduces challenges linked to its size, speed, diversity, and complexity. Addressing these issues requires the design of advanced and efficient techniques for data processing and analysis. This study proposes an enhanced clustering approach that integrates the Firefly Algorithm(FA) with the K-Means method to improve clustering accuracy and scalability in large-scale data environments. The proposed hybrid algorithm was implemented and evaluated in a multi node Apache Hadoop cluster environment on AWS EC2. Experiments were conducted on the PokerHand and Tabular Playground datasets. Performance was assessed using clustering purity, Davies Bouldin Index (DBI), and computational time. Results demonstrate that the hybrid method outperforms traditional clustering techniques. The results show that better clustering purity, lower DBI values, and significant gains in execution time and resource utilization in a distributed setting. These findings confirm the effectiveness of combining metaheuristic optimization with clustering for large-scale data categorization, paving the way for real-time analytics. Future work will focus on refining the hybrid model, exploring alternative optimization strategies, and extending its application to other big data frameworks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Scalable Data Clustering Using Firefly Algorithm in Distributed Environment

  • Shivalingappa Battur,
  • N. Tejas,
  • B. Naveenkumar,
  • K. Aditi,
  • Trupti V,
  • S. G. Totad

摘要

In today’s digital transformation, big data plays a major role in fostering innovation and guiding strategic direction. Big data offers many opportunities; it also introduces challenges linked to its size, speed, diversity, and complexity. Addressing these issues requires the design of advanced and efficient techniques for data processing and analysis. This study proposes an enhanced clustering approach that integrates the Firefly Algorithm(FA) with the K-Means method to improve clustering accuracy and scalability in large-scale data environments. The proposed hybrid algorithm was implemented and evaluated in a multi node Apache Hadoop cluster environment on AWS EC2. Experiments were conducted on the PokerHand and Tabular Playground datasets. Performance was assessed using clustering purity, Davies Bouldin Index (DBI), and computational time. Results demonstrate that the hybrid method outperforms traditional clustering techniques. The results show that better clustering purity, lower DBI values, and significant gains in execution time and resource utilization in a distributed setting. These findings confirm the effectiveness of combining metaheuristic optimization with clustering for large-scale data categorization, paving the way for real-time analytics. Future work will focus on refining the hybrid model, exploring alternative optimization strategies, and extending its application to other big data frameworks.