The paper describes the performance of parallel NetCDF output in the INM RAS Earth system model (INMCM). An output library with a compatible interface was developed for the INMCM that allows switching between the original serial output to direct unformatted files and the parallel implementation based on the NetCDF Fortran library. We evaluate the performance of several approaches to organize the data for NetCDF output, namely, storing each variable in a separate file, storing all variables of the same shape in a single file and batching the output by time. The experiments were carried out on two HPC systems that differ in size, interconnect network and storage file system. To study how data volume affects the output performance we considered two versions of INMCM with different spatial resolutions. The NetCDF outperforms the original serial implementation when writing large amounts of output data, but can be slower when writing a large number of small files. Merging 2D variables into a single NetCDF file is crucial to reach the performance of the serial implementation. Time batching technique has proven to be a powerful method to increase the data output performance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Performance of Parallel NetCDF Output in the INM RAS Earth System Model

  • Maria Tarasevich,
  • Ivan Tsybulin,
  • Vasilisa Bragina,
  • Evgeny Volodin

摘要

The paper describes the performance of parallel NetCDF output in the INM RAS Earth system model (INMCM). An output library with a compatible interface was developed for the INMCM that allows switching between the original serial output to direct unformatted files and the parallel implementation based on the NetCDF Fortran library. We evaluate the performance of several approaches to organize the data for NetCDF output, namely, storing each variable in a separate file, storing all variables of the same shape in a single file and batching the output by time. The experiments were carried out on two HPC systems that differ in size, interconnect network and storage file system. To study how data volume affects the output performance we considered two versions of INMCM with different spatial resolutions. The NetCDF outperforms the original serial implementation when writing large amounts of output data, but can be slower when writing a large number of small files. Merging 2D variables into a single NetCDF file is crucial to reach the performance of the serial implementation. Time batching technique has proven to be a powerful method to increase the data output performance.