Pipeline parallelism has become a critical strategy for training ultra-large language models, yet it faces memory bottlenecks in early pipeline stages due to memory imbalance. While prevalent techniques like swapping address this issue, they often fail to balance overhead avoidance with significant memory savings. We propose LPipe, an efficient pipeline training system that achieves memory balance on low inter-GPU bandwidth servers through bubble-filled activation swapping. LPipe employs temporal bubble-filled swapping to maximize the utilization of temporal communication bubbles, extending the overlapped transfer time for data swapping and reducing communication overhead. To further optimize communication efficiency, LPipe utilizes memory-reused parallel swapping, enabling parallel execution of eviction and loading operations without increasing peak memory usage, thereby fully leveraging the bidirectional bandwidth of communication links and masking additional communication overhead. Evaluations on Llama-2 and GPT-3 models of varying sizes, conducted on the A800 GPU server without NVLink, demonstrate that LPipe achieves throughput improvements of up to 1.32 \(\times \) compared to widely used recomputation baselines.