Efficient storage and retrieval of large-scale scientific data remain critical challenges in HPC environments, especially as heterogeneous architecture and evolving system complexities continue to increase demands on application developers. This work-in-progress paper investigates the potential of in-memory computing file systems, specifically Ignite’s Distributed File System (IDFS), for mitigating I/O bottlenecks in scientific workflows. We evaluate IDFS performance by single-node deployment at the NERSC Perlmutter supercomputer for latency in data access, throughput, and resource utilization. Our preliminary results prove the feasibility of using IDFS to improve I/O efficiency in large-scale scientific applications. This work adds to the current efforts in scalable data management and performance optimization, hence setting the base for future multi-node evaluations and integration with advanced parallel and distributed computing methodologies.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Accelerating I/O in Scientific Workflows with the Impact of Apache Ignite’s In-Memory File System

  • Vijayalakshmi Saravanan,
  • Sai Karthik Navuluru,
  • Lakshman Tamil,
  • Khaled Ibrahim

摘要

Efficient storage and retrieval of large-scale scientific data remain critical challenges in HPC environments, especially as heterogeneous architecture and evolving system complexities continue to increase demands on application developers. This work-in-progress paper investigates the potential of in-memory computing file systems, specifically Ignite’s Distributed File System (IDFS), for mitigating I/O bottlenecks in scientific workflows. We evaluate IDFS performance by single-node deployment at the NERSC Perlmutter supercomputer for latency in data access, throughput, and resource utilization. Our preliminary results prove the feasibility of using IDFS to improve I/O efficiency in large-scale scientific applications. This work adds to the current efforts in scalable data management and performance optimization, hence setting the base for future multi-node evaluations and integration with advanced parallel and distributed computing methodologies.