Parallelizable Loop Detection using Pre-trained Transformer Models for Code Understanding

Parallel programming is essential to utilize multi-core processors but remains challenging because it requires extensive knowledge of both software and hardware. Various automatic parallelization tools based on static analysis have been developed to ease the development of parallel programs. However, hand-parallelized codes still outperform auto-parallelized codes. Meanwhile, transformer-based large language models have made ground-breaking progress in coder understanding and generation tasks. In this paper, we fine-tune a transformer-based code understanding model, CodeT5, to create a model for automatically identifying parallelizable for-loops. The trained model helps developers to identify independent for-loops that can be potentially parallelized using tools such as OpenMP to improve the program performance. Our model is trained over 90,908 for-loops collected from 9 million C/C++ source files of public GitHub repositories, and achieves a 0.895 F1 score in identifying parallelizable for-loops in public GitHub projects and a 0.713 F1 score in the NAS Parallel Benchmark suite.