A distributed skewed stream processing system based on scoring high-frequency key perception
摘要
The onset of the big data era has ushered in a proliferation of stream-based applications across diverse domains. Nevertheless, the real-world streams are characterized by significant skewness. This attribute skew, in turn, leads to workload imbalances within distributed stream processing systems (DSPSs), resulting in increased response times and diminished throughput in DSPS applications. Building upon this challenge, we present the design and implementation of a distributed skewed stream processing system named SH-Stream, leveraging Scoring High-frequency (SH) key perception. SH-Stream comprises two integral components: the Stream Data Prediction Module and the Efficient Stream Schedule Module. To accurately identify high-frequency keys in streams, we introduce a scoring probabilistic prediction algorithm and establish effective maintenance mechanisms for these high-frequency keys. Load balancing is achieved through the judicious splitting of identified high-frequency keys. We implement and assess this functionality across multiple stream processing platforms, employing large-scale synthetic and real-world datasets for comprehensive evaluation. Our experimental results reveal a notable improvement in throughput, with an impressive 89.9% and 27% enhancement compared to KG and PKG, respectively. Additionally, latency experiences a substantial