Scholarlink

<p>Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system’s performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82<InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="10586_2025_5108_Article_IEq1.gif" Format="GIF" Height="13" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">\(\times\)</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36<InlineEquation ID="IEq2"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="10586_2025_5108_Article_IEq1.gif" Format="GIF" Height="13" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">\(\times\)</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Nime: a native in-memory compute framework for cluster computing

  • 期刊论文

  • 发表日期：

    2025年7月31日

查看全文
  • Chao Chen,
  • Zhenghua Wang,
  • Chen Jiang,
  • Zheng Wang

摘要

Due to the increasing demand for cluster computing, various data analytics frameworks have been proposed and Apache Spark is a widely used open-source framework. It divides the program into various tasks and leverages executors on different machines for parallel task processing. However, executors run on top of Java virtual machines (JVMs), which incurs a significant runtime overhead in terms of memory and compute resources and thus deteriorates the system’s performance. In this paper, we present NIME–a native in-memory compute framework for cluster computing–that aims to perform parallel task processing using native executors. The key idea is that NIME starts off with native manager and worker processes without JVMs. In addition, a dedicated scheduler combines data partitions for efficient processing without interruptions and a cached is leveraged for iterative computations. We evaluate the effectiveness of NIME on a compute cluster using the HiBench benchmark suite and compare the results with those from the Spark framework. Evaluation results indicate that compared to Spark, on average NIME achieves a 6.82 \(\times\) × speedup, while simultaneously reducing the memory usage by 84.69%. In addition, the execution speedup and memory reduction can reach up to 12.36 \(\times\) × and 93.97%, respectively. Together with an in-depth analysis, we show that by discarding the JVM, NIME significantly accelerates task executions and minimizes the compute and memory resource overheads.