Sync5D: Novel View Synthesis from a Single Image with 5D Consistency
摘要
Novel view synthesis from a single image is a critical yet challenging task in computer vision, often as an intermediate step in many algorithms such as 3D generation. Existing approaches employ generative models to synthesize the occluded content under novel views while enforcing 3D constraints to maintain multi-view consistency. Although significant progress has been achieved, current methods only focus on geometry consistency, resulting in inadequate modeling of 5D scene properties (spatial location (x, y, z) and viewing direction \((\theta , \phi )\) ) and suboptimal generation of view-dependent effects (e.g., specular highlights). Therefore, we present Sync5D, a novel diffusion-based approach for novel view synthesis from a single image that ensures both plausible view-dependent effects and multi-view consistency. The core motivation is to lift the constraint of the generative model from 3D to 5D and explicitly model the view-dependent features through three key contributions: (1) We conduct diffuse-specular channel separation during the denoising process for explicit view-dependent feature handling. (2) We design a dedicated module with view-encoding to provide pixel-level fine-grained view-dependent guidance for the specular channel. (3) We curate Objaverse-Shiny, an Objaverse subset with enriched concentration of view-dependent data, for training and evaluation. Experimental results demonstrate that our approach achieves better benchmarks on both scanned and synthesis datasets, outperforms existing methods in generating realistic view-dependent effects, and preserves superior multi-view consistency.