Accelerating Distributed Graph Partitioning Based on Intelligent Network
摘要
Modern graph-structured data from social networks with billions of edges to biological systems exceeding trillions of nodes—has driven exponential growth in both scale and complexity. Traditional distributed graph partitioning frameworks like Hadoop and Spark, however, struggle to manage this heterogeneity due to static partitioning strategies that disrupt graph topological structures. This results in excessive cross-node communication, with shuffle phase consuming over 60% of runtime for witter-scale graphs. The emergence of programmable data planes enables an approach. Our proposed GraphNet system addresses these challenges through three innovations: a GraphNet protocol using programmable switches to encode edge metadata and cache vertex-edge structures for efficient data exchange; dynamic heterogeneous partition expansion that adapts to real-time node resource metrics; and intelligent partition migration to minimize communication overhead. By unifying network aware optimization with distributed computing, GraphNet reduces communication overhead by at least 40% and improves partitioning efficiency by 1.3–6x compared to traditional methods.