WindowView: A Stream Processing Engine for Big-Data Analytics
摘要
Modern real-time streaming analytics typically involves complex workloads—such as machine learning and graph processing—that demand significantly more CPU resources than traditional stream processing tasks. However, most state-of-the-art stream processing engines (SPEs) fail to fully leverage modern hardware, thus resulting in reduced data processing throughput, particularly for CPU-intensive workloads like data analytics. This low throughput often leads to record accumulation, and the resultant inability to process records may promptly increase system latency. This paper presents a novel stream processing engine, WindowView, which prioritizes high-throughput analytics as a core design goal. WindowView achieves high-throughput processing through a columnar execution engine combined with vectorized optimization. To further minimize the overhead of state operations, we design a specialized storage engine for caching intermediate window states. Experimental results demonstrate that WindowView outperforms Apache Flink by 14.8 \(\times \) on the Yahoo! Streaming Benchmark. Moreover, considering a representative analytics workload using the New York City Taxi dataset, WindowView surpasses both Apache Flink and Apache Spark Streaming by up to 2700 \(\times \) .