Federated Aggregated Micro Pipelines (FAMP) for Efficient Data Processing in MongoDB: A Case Study on Universities and Movie Dataset

MongoDB is a widely used NoSQL database that provides a customizable structure and is simple to move both vertically and horizontally to enhance outcomes. It is a document-oriented database that scales horizontally by distributing data across multiple computers using the auto-sharding technique rather than storing or retrieving data as tables. The aggregation pipeline in MongoDB and how well it handles processing a lot of data are the main topics of this paper. Through a webhook produced by MongoDB, a movie search engine is built using the federated dataset Sample Mflix Movies. Additionally, a federated database and aggregation pipeline were built using the datasets for courses and universities, and the results were displayed. The main contribution of this paper is the demonstration of Federated Aggregated Micro Pipelines (FAMP), a novel approach to efficient data processing in MongoDB. FAMP is a distributed processing framework that allows for efficient data processing by breaking up operations into microservices and aggregating the results. The FAMP system provides scalability, performance, and flexibility to MongoDB, making it an ideal solution for analyzing large datasets. Overall, this paper showcases the benefits of using MongoDB and FAMP for data processing, particularly in the context of a Movies and universities dataset [1]. The results demonstrate the effectiveness of FAMP in producing valuable insights and improving efficiency in data processing.