An optimization approach for big data processing in complex networks fusing graph computing and spark architecture
摘要
This paper presents an optimization approach for big data processing in complex networks by integrating graph computing and Spark architecture. The paper addresses key challenges including deep integration of graph computing with Spark, accurate representation of complex network data, and dynamic network modeling for different scenarios. The proposed methodology comprises three main components: a novel graph partitioning and communication optimization strategy that balances data volume and computational load while reducing overhead, a graph neural network (GNN)-based representation learning framework that captures multi-scale information and complex relationships within networks, and dynamic network models with specialized analysis indicators that capture real-time network changes and provide scenario-specific insights. Experimental results demonstrate the effectiveness of the approach, showing significant improvements in processing time, communication overhead, load balance, and model accuracy compared to baseline methods. Ablation studies and parameter analyses further confirm the contribution and practical applicability of each component. The proposed work offers a robust solution for efficient big data processing in complex networks, providing valuable insights for related research and real-world applications.