A transformer network for video inpainting with optical flow guidance and spatio-temporal decoupling
摘要
Transformer networks are effective in video processing, but in video inpainting, occlusions often cause weak feature representations, resulting in query degradation and attention errors. To address this issue, we propose a spatio-temporal inpainting framework that establishes a complete optical flow-guided pipeline, covering dynamic region recognition, spatio-temporal modeling, and video restoration. First, we design a dynamic-aware convolutional network guided by optical flow, which generates soft masks for dynamic regions based on optical flow consistency residuals and edge responses, and employs multiple structural loss functions for precise modeling. Next, we introduce a solution to the query degradation problem within the transformer architecture by incorporating a confidence-aware query construction module and a guidance-constrained attention module. These modules enhance the semantic representation and attention distribution of query vectors in the spatial and temporal dimensions, respectively. Furthermore, we develop a transformer-based architecture with spatio-temporal decoupling guided by optical flow. The temporal branch introduces an occlusion-aware multi-head attention mechanism, which performs cross-frame alignment using optical flow offsets and occlusion masks. The spatial branch adopts a cross-scale shifted window attention module to improve structural representation through multi-scale windows and offset strategies. Finally, a combined spatio-temporal reconstruction loss and frequency-domain consistency loss are employed to enhance structural accuracy and temporal continuity. Experiments show that the proposed method outperforms existing video inpainting models in both visual quality and quantitative metrics.