Event-Based Data Pipelines in Recommender Systems: The Data Engineering Perspective

Recommender Systems (RS) are information retrieval systems that can be used for serving personalized content to online users. Most industrial recommendation systems utilize a large amount of online data to generate personalized recommendations for users. The quality of the data plays an important role in the performance of the RS. The majority of the RS data is generated from event data that are stored in data lakes through multiple data pipelines. Event-based data pipelines have emerged as a popular approach to handle the massive amount of data generated by modern applications. In this paper, we explore the impact of event-based data pipelines on recommendation systems. We discuss how these pipelines enable efficient data ingestion, real-time processing, and low-latency recommendations.