LMStor: Storage Acceleration Design for Large Models

Recently, large-scale and diverse training datasets have become the key to superior semantic understanding capabilities of large models. As the number of parameters grows, large models have increasingly higher requirements for data volume during training, which poses challenges to traditional storage systems. This leads to problems such as high overhead for checkpoint fault tolerance, slow data loading that hampers computational efficiency, and significant consumption of storage resources. To address these issues, we propose a storage acceleration design for large models, LMStor, to improve training efficiency and storage space utilization of large models. Experimental results demonstrate that LMStor reduces checkpoint file saving time by 92.99%, increases the loading operations per second (OPS) of small files by 7.46 times, and shortens the data loading process by 86.6%. Furthermore, LMStor effectively conserves storage resources.