Towards Locality-Aware Host-to-Device Offloading in OpenMP

The computational demand from scientific and industrial applications has grown significantly, driven by advances in scientific simulations across various fields such as climate forecasting, molecular dynamics, and medicine. This has led to a shift from purely CPU-based systems to heterogeneous architectures that include CPUs and accelerators like General Purpose Graphic Processing Units (GPGPUs), enabling faster computations by offloading demanding tasks to these accelerators. Modern High Performance Computing (HPC) systems nowadays feature Non-Uniform Memory Access (NUMA) designs and are additionally equipped with multiple GPUs that are attached to different parts or sockets of the system. Considering the underlying architecture and proximity between CPU cores on the host and of GPUs during offloading is crucial to achieve good performance. Current programming models like OpenMP lack locality-aware device selection capabilities, necessitating manual code optimization for specific architectures. Our research addresses these issues by exploring CPU-to-GPU offloading performance, suggesting locality-aware host-to-device API extensions for OpenMP, and incorporating these extensions into the LLVM OpenMP runtime. Evaluations on a system with four GPUs demonstrate significant performance improvements through locality-aware device selection.