Using MPIs Non-Blocking Allreduce for Health Checks in Dynamic Simulations

Large-scale simulations often require frequent checks on global conditions that are not directly needed for the computation itself. While such health checks are not an integral part of the numerical algorithm, they serve an important role in controlling and coordinating the simulation. In distributed parallel computations their required communication may negatively impact the actual parallel computation and result in unnecessary synchronization points. We show that by using a non-blocking reduction these synchronization requirements can be loosened and the impact on the actual computation minimized. Further, it enables us to shift the communication into the background and progress it during the MPI calls that are done during the computation anyway. We demonstrate that a sufficient amount of MPI calls in between is required to allow for progress to happen. The presented approach delays the decision to be made in response to those health checks. But as it is not vital for the correct computation itself such a delay is usually tolerable and could offer a more robust scaling to large process counts.