Cost-Efficient SQL Engines for Big Data Queries Based on EMR Clusters
摘要
This study provides a cost-focused evaluation of SQL query engines operating within a Big Data architecture using Amazon EMR clusters. It aims to determine the most efficient option for executing queries in disconnected processing scenarios, comparing Hive, Spark, and Presto on EMR with the three execution modes offered by Amazon Athena: default (measured by terabytes scanned), provisioned capacity, and Apache Spark runtime. The evaluation is based on the TPC-DS benchmark on the 1TB and 3TB scales, with data stored on Amazon S3 to maintain consistency between platforms. Six of the most computationally demanding TPC-DS queries –identified in prior studies– were executed on each engine. Performance was measured in terms of the total query execution time and cost, the number of nodes in the EMR clusters, the types of instances, and the duration of the run. All EMR clusters were deployed on-demand and shut down after processing through automatic scripts to simulate ephemeral disconnected usage. Among the technologies tested, Spark on EMR consistently delivered the lowest cost per query and the fastest execution time at larger data volumes. It also successfully ran a query that none of the other engines, including Athena, could complete. Although Hive and Presto performed reliably, they incurred higher costs and overhead. Overall, the findings suggest that Spark on EMR is currently the most cost-effective solution for batch-style disconnected query processing where moderate execution time is acceptable. The guide and scripts used in this work are published in a GitHub repository for the next related studies. The source code, data, and/or other artifacts have been made available at https://github.com/marcelleonlafebre/experiments.