StreetSyn: A Full Radiance Field Solution for Street and Vehicle Free-View Synthesis
摘要
Starting from sparse views of real-captured scenes and a synthetic dataset of 3D vehicles, we aim to synthesize photo-realistic street views with moving vehicles, editable illumination, and controllable viewpoints which is a significant task for autonomous driving simulation. The problem is very challenging as only sparse views are available for recovering such a complex street environment. In this paper, we propose a full radiance field scheme for free-view synthesis of street scenes and vehicles. Benefiting from the scheme that both the scene and the vehicle are represented as radiance fields, illumination can be directly extracted from the real-captured scenes and transferred to the synthesized vehicle. The ambient illumination is modeled as a mixture of Spherical Gaussians (SGs) with different frequencies, which turns out to be effective in recovering the low-frequency sky illumination and high-frequency sun illumination. Experiments show that our model can synthesize street view and vehicle images in free views, and significantly outperforms previous works in photo-realism and lighting modeling accuracy.