PushNet: 3D reconstruction from a single image by pushing

Taking inspiration from the recent advancements in deep learning within the three-dimensional (3D) domain, we propose an end-to-end deep learning framework to reconstruct 3D shapes in point cloud format from a single color image. While many state-of-the-art learning-based 3D reconstruction methods are constrained to fixed resolutions, our framework, named PushNet, can produce point clouds with arbitrary resolutions and only require sparse point clouds during training. It predicts a push force for each randomly sampled spacial point and leads the point to project onto the surface of the underlying 3D object in the image. The network also employs a parallel design, allowing it to be trained on sparse point clouds and then generate point clouds of any resolution without degrading the quality or requiring any fine-tuning. Experiments on synthetic datasets and real datasets demonstrate the effectiveness of our method for inferring 3D shapes. We also demonstrate that our predicted point clouds can produce high-fidelity meshes after applying surface reconstruction algorithms. Experiments on linear interpolation, point cloud upsampling, and textured 3D reconstruction also prove the effectiveness of our framework.