3RE-Net: Joint Loss-REcovery and Super-REsolution Neural Network for REal-Time Video

Real-time video over the Internet suffers from packet loss and low network bandwidth. The receiving side may receive down-sampled video with damaged frames. In this work, we are motivated to enhance the quality of video by joint loss recovery and super-resolution. We propose Joint Loss-REcovery and Super-REsolution Neural Network for REal-time Video (3RE-Net), to recover the loss and super-resolve a damaged frame. 3RE-Net has two unprecedented advantages: (1) It only utilizes preceding frames and the current frame as input, as waiting for future frames causes additional delay, which is not suitable for real-time video streaming. (2) 3RE-Net induces small inference delay, which is applaudable for real-time videos. To mitigate computational workload, we jointly process the loss recover and super-resolve by reusing motions and features beneficial for both super-resolution and loss recovery. The design of 3RE-Net can be summarized as follows: It first extracts motions and features from the frames, and propagates the extracted motions and features through warping, synthesis, feature-level alignment, and deep detail refinement modules. Through this way, we can first obtain a set of warped candidate frames, which are later used to generate spatio-temporal consistent feature maps through synthesis and alignment. The output frame can be reconstructed by the feature maps in high resolution and loss free. We conduct experiments to compare 3RE-Net with state-of-the-art benchmark schemes. Results demonstrate that 3RE-Net outperforms all existing benchmarks in terms of both quality and delay.