Engineering Real-time Data Pipelines using Apache Kafka

So far we looked at batch processing pipelines. In this chapter, we will be looking at real-time streaming processing pipelines and a library that is incredibly rich in functionalities. We are going to discuss Apache Kafka. Apache Kafka is a distributed and fault-tolerant streaming and messaging platform. Kafka helps build event streaming pipelines that can capture creation of new data and modification of existing data in real time and route it appropriately for various downstream consumption purposes. In this chapter, we will look at Kafka, its architecture, and how Kafka can help build real-time data pipelines.