Dual Dreamer: Extending Single-View Dreamer with Few Shot of Complementary Views

Recently, diffusion models demonstrate the potential of generating consistent novel views from a single view. Due to the limited object information contained in a single input view, controlling generation of out-of-sight views becomes challenging. In this work, we propose Dual Dreamer, a method of generating novel views with complementary views. Specifically, we focus on fine-tuning the single-view input diffusion model to extract complementary information from additional known views and combine it with the original information for novel view synthesis (NVS). To achieve this, Dual Dreamer introduces a 3D attention mechanism to enable the model to learn common features of the complementary views and enhance information exchange. By utilizing view difference between predefined and target views as weights, predefined views are fused as priors to enhance the consistency of multi-view synthesis. Finally, in order to avoid consuming resources on large-scale datasets, the proposed Dual Dreamer model is trained in a similar way like ControlNet, enabling efficient training on small datasets. Extensive evaluation on various datasets demonstrates that our approach achieves consistent, high-quality generation results compared to methods that rely on a single input view.