Lane detection is a crucial task in autonomous driving. Single-frame lane detection methods exhibit suboptimal performance in various scenarios, thus video instance segmentation methods are adopted in lane detection. However, existing video instance lane detection methods also face challenges. The intricate design of the network leads to inefficient utilization of computational resources, and insufficient consideration of the lanes’ distinctive characteristics results in imprecise segmentation performance. To address these issues, we propose Integrated Memory Optimization Network (IMO-Net), a novel and efficient network based on space-time memory network (STM), to reinforce the effectiveness of video instance lane detection. We introduce the temporal scaling shift aggregation (TSSA) mechanism and the CA-Gru module to enhance the network’s efficiency and ability to handle long-term memory features. Specifically, compared to general objects, lane lines exhibit sparsity across the entire image while also containing abundant local details. To amplify the model’s aptitude for extracting the local detail and global information of spatio-temporal memory, we design the core memory aggregation (CMA) module, which enhances extracting essential spatio-temporal memory. Experimental results demonstrate that our method achieves state-of-the-art performance on the VIL-100 dataset and performs well on Tusimple dataset.