Graphics and Vision’s Camera Calibration and Applications to Neural Radiance Fields

Pre-training and fine-tuning are important tasks for developing deep-learning models in many modals, e.g., image, video, and text. Recently proposed works enable Neural Radiance Fields (NERF) ready for using with those two tasks. In combination with training and tuning, techniques for enriching data, for example, data augmentation and masking, are also crucial for making deep-learning successfully. Inspired by the field of computer vision (e.g., classification, detection, segmentation, and so on) where the data augmentation have been performed online for each input batch; in NERF, we plan to use computer graphics’ libraries (especially, OpenGL) to generate views online with virtually posed cameras. Practically, OpenGL has a camera frame; however, this kind of cameras is configured differently to the one in computer vision. Specifically, OpenGL’s cameras do not have intrinsic camera matrix as in computer vision. This paper presents an approach to calibrate cameras in the two fields, i.e., computer vision and OpenGL. The developed technique enables to generate views online that can be integrated naturally to NERF’s research works. Utilizing fundamental features of graphics pipelines, e.g., geometric transformation, lighting, shading and texturing, the proposed technique is able to support similar operations in the data augmentation for images, video and text. Experiments in this paper shows that: (a) the intrinsic camera matrix can be loaded into the projection matrix in OpenGL; (b) the intrinsic camera matrix used by the proposed technique can be successfully recovered by COLMAP [1], which means that our calibration method is workable; and (c) the generated views can used with nerfstudio [2] to generate 3D-models, which means that the proposed method can generate views that are compatible with NERF’s algorithms.