Tall-and-Skinny QR Factorization for Clusters of GPUs Using High-Performance Building Blocks

We build a message-passing realization of the QR factorization for tall-and-skinny matrices on top of highly parallel linear algebra kernels, such as various types of matrix multiplications and triangular system solves, plus a few small Cholesky decompositions. Our solution, using either the NVIDIA Collective Communications Library (NCCL) or a plain instance of MPI as the message-passing layer, and the implementation of these kernels in linear algebra libraries, can run both on clusters of multicore nodes, possibly accelerated with GPUs, or on multi-GPU platforms. The experimental evaluation of our parallel algorithm for the QR factorization on a cluster of 8 nodes with NVIDIA A100 boards shows significant acceleration factors over a code from MAGMA, based on Houselholder reflectors, that provides the same functionality. In addition, the experiments show a fair weak scalability when the problem has many more rows than columns.