Data analytics is an inevitable component for any major organization that works with data, including banks, service aggregators, governments, insurance providers, and advertising companies. There are two primary approaches to parallelizing a data analytics engine: by requests (serial execution within a request) or by data (serializing and distributing requests). Running too many requests simultaneously can exceed memory bandwidth limits, while dedicating an entire machine to a single request may be inefficient due to data size and underutilization of processor cores. Can we combine these approaches to overcome their limitations—processing multiple requests concurrently while enabling data parallelism within each request? We evaluate composable threading solutions using the New York City Taxi benchmark to address this question.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Composable Parallelism in Data Processing Tasks

  • Vladimir Bakhtin,
  • Nikita Kataev,
  • Alexander Kolganov,
  • Dmitry Zakharov,
  • Alexander Smirnov,
  • Anton Malahov

摘要

Data analytics is an inevitable component for any major organization that works with data, including banks, service aggregators, governments, insurance providers, and advertising companies. There are two primary approaches to parallelizing a data analytics engine: by requests (serial execution within a request) or by data (serializing and distributing requests). Running too many requests simultaneously can exceed memory bandwidth limits, while dedicating an entire machine to a single request may be inefficient due to data size and underutilization of processor cores. Can we combine these approaches to overcome their limitations—processing multiple requests concurrently while enabling data parallelism within each request? We evaluate composable threading solutions using the New York City Taxi benchmark to address this question.