Data Lake Optimization: An Educational Analysis Case

This study focuses on enhancing the performance of Universidad Internacional SEK’s (UISEK) Data Lake by addressing challenges in computing resource consumption from a prior data lake implementation. Notably, data has been sourced from the Canvas Learning Management System based on the university’s usage since 2019 for both implementations. The restructuring, carried out through three layers, successfully mitigated previous computing resource challenges. Following the CRISP-DM framework, the new approach exhibited substantial improvements over the previous version. Results include a 73.1% reduction in the estimated size of the last dump, 51.7% more efficient storage utilization in the user behavior table, and a 4.3% improvement in CPU consumption. Additionally, showcased a 50% reduction in ingestion time. The study emphasizes the significance of a well-organized Data Lake governance structure for streamlined data management. The presented improvements lay a solid foundation for future analyses and machine learning models. Moreover, the study underscores the role of automated processes in maintaining an updated Data Lake, ensuring its relevance for decision-making at UISEK. Overall, this work contributes to advancing the efficiency and performance of educational Data Lakes, providing valuable insights for decision-making and continuous enhancement of the educational experience.