Real-Time Intelligence with Spark Structured Streaming

Many people think of streaming as some very low-latency continuous real-time events like X, formerly Twitter, feeds or IoT devices. While that was the original use case, streaming has evolved over the years to allow integration with other non-real-time tables and as a useful technique to enable incremental processing for batch pipelines. In this chapter, we will first go back in time to visit Spark Streaming; then we will look at the latest Databricks Structured Streaming engine. They are largely the same, but Structured Streaming has better abstractions. Then we will look at how to use Lakeflow Declarative Pipelines to process streaming. Apache Spark offers two popular streaming processing engines: Spark Structured Streaming and real-time mode. While both engines are designed for real-time data processing, they have distinct advantages and use cases.