CVSR: complex-valued networks for video super-resolution
摘要
In video super-resolution (VSR), bidirectional sequences (i.e., forward and backward sequences) inherently describe the temporal flow and provide oriented motion information for high-resolution reconstruction. However, few works consider further modeling of bidirectional features, and they only aggregate these forward features and backward features into a group of ordinary feature maps via concatenation. Therefore, the correlation between the two features in opposite directions remains to be explored. In this paper, we provide a novel viewpoint for VSR, which constructs a higher relationship with respect to the forward features and backward features in the complex number domain. We respectively treat a forward feature and a backward feature as the real component and the imaginary component of a complex number and compute them according to the rules of complex number. This computation allows the bidirectional features to embrace a more coupling relation, so as to excavate high-dimension information used for reconstruction. Specifically to achieve complex VSR framework, we present two transplantable modules, i.e., complex transform and complex attention. The complex transform aims to generally change the bidirectional features to complex features by constructing a complex feature extraction. The complex attention can adaptively rectify the complex features for each complex feature map, so as to project a feature map to a specific frequency. In addition, we analyze the superiority of complex representations compared with real-valued representations. Extensive experiments on representative bidirectional networks (BasicVSR and IconVSR) and their improved form (IBRN) show the effectiveness and robustness of complex VSR framework. Codes are available at https://github.com/ttcluo/CVSR.git.