Wavelet Residual Learning for Efficient Future Frame Prediction from Natural Video Sequences

Video prediction is a challenging task of predicting the future frames based upon the past frames. Existing deep learning (DL) based methods either directly hallucinate the pixel values in high-dimensional video space, resulting in blurry predictions, or decompose the input space into lower-dimensional intermediate representations requiring additional assumptions. Moreover, due to the complexity of the task, existing methods often propose complex networks with high memory and compute requirements. To address these limitations, we propose a simpler residual based architecture in wavelet domain for faster and accurate video prediction. The natural sparsity of wavelet domain makes the learning task easier for the model. To the best of our knowledge, this is the first DL-based method that predicts future frames entirely in the wavelet domain. Our approach takes 2-dimensional Discrete Wavelet Transform (2D-DWT) sub-bands of video frames as input and learns to infer the difference between the wavelet coefficients of the adjacent frames (Temporal Wavelet Residuals). Final prediction is obtained by adding the input to the predicted residuals followed by application of Inverse Discrete Wavelet Transform (IDWT). The sparsity of wavelet residuals reduces the training and inference time. Extensive experimentation demonstrates that the proposed approach is computationally efficient and still competitive with the state-of-the-art methods both qualitatively and quantitatively, on KTH and KITTI datasets.