<p>Pipeline parallelism is a distributed method used to train deep neural networks and is suitable for tasks that consume large amounts of memory. However, this method entails a large overhead because of the dependency between devices for performing forward and backward steps using multiple accelerator devices. Although a method to remove forward step dependency through the all-to-all approach has been proposed for training compute-intensive models, it incurs a large overhead when training with many devices and is inefficient with respect to weight memory consumption. Alternatively, we propose a pipeline parallelism method that reduces both network communication using a self-generation concept and overhead by minimizing the weight memory used for acceleration. In a DarkNet53 training throughput experiment using six devices, the proposed method outperforms a baseline by approximately 63.7% in reduction of overhead and communication costs and achieves less memory consumption by approximately 17.0%.</p>