The management of large-scale data in high-performance computing (HPC) environments requires effective storage systems. Distributed and parallel file systems, such as Lustre, offer solutions but are often complex and challenging for smaller projects. This paper explores the use of JULEA, a user-space framework, to prototype new storage management approaches. Object storage, with its simplicity and flexibility, is evaluated as a viable alternative. We compare three object storage clients—OpenIO, AWS, and MinIO—in terms of efficiency, compatibility, and usability. While OpenIO presented significant integration challenges, MinIO and AWS clients were successfully implemented. The study reveals that the AWS C++ SDK outperforms MinIO in terms of performance and stability, making it the preferred choice for JULEA backend development. Performance evaluations demonstrated AWS’s superior read and write capabilities and better documentation and community support. This paper concludes that AWS offers the most robust solution for developing a JULEA object storage backend, recommending its adoption for future projects.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Comparative Analysis of Object Storage Clients for JULEA Framework: AWS vs. MinIO

  • Ram Charan Tej Grandhe

摘要

The management of large-scale data in high-performance computing (HPC) environments requires effective storage systems. Distributed and parallel file systems, such as Lustre, offer solutions but are often complex and challenging for smaller projects. This paper explores the use of JULEA, a user-space framework, to prototype new storage management approaches. Object storage, with its simplicity and flexibility, is evaluated as a viable alternative. We compare three object storage clients—OpenIO, AWS, and MinIO—in terms of efficiency, compatibility, and usability. While OpenIO presented significant integration challenges, MinIO and AWS clients were successfully implemented. The study reveals that the AWS C++ SDK outperforms MinIO in terms of performance and stability, making it the preferred choice for JULEA backend development. Performance evaluations demonstrated AWS’s superior read and write capabilities and better documentation and community support. This paper concludes that AWS offers the most robust solution for developing a JULEA object storage backend, recommending its adoption for future projects.