Intelligent EcoETL: An Energy-Aware Framework for Sustainable Extract-Transform-Load Operations in Big Data Processing

In the rapidly expanding domain of big data processing, traditional Extract-Transform-Load (ETL) pipelines impose substantial energy demands on computing infrastructure, contributing to increased carbon emissions and operational costs in data centers. This paper introduces EcoETL, a novel intelligent energy aware ETL framework designed to investigate adaptive re-source optimization strategies for sustainable data processing. Built on Py-thon using pandas and psutil libraries, EcoETL features adaptive decision logic that dynamically assesses system constraints file size, estimated row count, memory usage, and available memory to switch between eco mode and speed mode processing approaches. The framework incorporates three innovations: an EnergyMonitor module with high frequency sampling for re-al time resource tracking, an OperationOptimizer that reorders ETL pipelines to prioritize data reducing operations, and comprehensive validation mechanisms for processing integrity. Experimental evaluation using NYC Taxi datasets reveals significant insights into the relationship between dataset characteristics and optimization effectiveness. The framework demonstrates competitive performance for large datasets exceeding 400 MB under memory constrained conditions, while smaller datasets experience substantial optimization overhead outweighing energy efficiency benefits. Statis tical validation across multiple test runs confirms measurement reliability and identifies critical performance boundaries determining framework applicability. The EnergyMonitor achieved high accuracy in resource tracking, while the OperationOptimizer contributed measurable improvements when processing conditions aligned with framework strengths. The research contributes empirical evidence regarding scale dependent ETL optimization, validates adaptive monitoring strategies for energy aware computing, and identifies infrastructure dependencies influencing sustainable data processing. Results indicate optimization benefits scale with data complexity while framework overhead remains constant, suggesting algorithmic refinements could enhance deployment viability. This work advances understanding of energy aware data pro-cessing challenges while establishing performance boundaries for sustainable ETL implementation in re source constrained environments, providing foundation for energy efficient solutions aligned with sustainable computing objectives.