Python is the dominating data science language, leaving behind other languages like C++, Java and R. Python libraries wrap highly tuned, efficient, accurate C++ and C code for linear algebra, numerical methods and data manipulation. Moreover, the Python runtime works flawlessly across diverse operating systems (Linux, Windows) and CPU architectures, including x86 and ARM. From an accelerator perspective, Python code is processed on multi-core CPUs and GPUs, whose power is not fully exploited. In this paper, we study how to improve Python I/O bottlenecks. We focus on data set summarization to compute a model on a large data set, stored on a CSV file (the most common format used in practice). Heeding these challenges, we introduce two simple, but fundamental, I/O optimizations: parallel multi-threaded read and chunk-based scan (similar to reading file blocks in a DBMS). An experimental validation on different cloud servers, provides a realistic scenario. We show our optimized Python code can work faster than existing Python functions, it exhibits almost linear speed up I/O as more threads are used (up to a limit), but it can still leverage parallel processing for the CPU-intensive floating point computations. To round up our study, we justify chunk size is a critical performance parameter that depends on data set size as well as cloud server configuration.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Accelerating Python Code with Parallel I/O

  • Robin Varghese,
  • Hashirul Quadir,
  • Ladjel Bellatreche,
  • Carlos Ordonez

摘要

Python is the dominating data science language, leaving behind other languages like C++, Java and R. Python libraries wrap highly tuned, efficient, accurate C++ and C code for linear algebra, numerical methods and data manipulation. Moreover, the Python runtime works flawlessly across diverse operating systems (Linux, Windows) and CPU architectures, including x86 and ARM. From an accelerator perspective, Python code is processed on multi-core CPUs and GPUs, whose power is not fully exploited. In this paper, we study how to improve Python I/O bottlenecks. We focus on data set summarization to compute a model on a large data set, stored on a CSV file (the most common format used in practice). Heeding these challenges, we introduce two simple, but fundamental, I/O optimizations: parallel multi-threaded read and chunk-based scan (similar to reading file blocks in a DBMS). An experimental validation on different cloud servers, provides a realistic scenario. We show our optimized Python code can work faster than existing Python functions, it exhibits almost linear speed up I/O as more threads are used (up to a limit), but it can still leverage parallel processing for the CPU-intensive floating point computations. To round up our study, we justify chunk size is a critical performance parameter that depends on data set size as well as cloud server configuration.