Accurate full-body pose estimation plays a key role in enhancing the virtual reality interaction experience. Existing single-source solutions face some limitations, e.g. undesirable accuracy due to sensor sparsity (the sensor-based) and pose distortion caused by camera perspective (the vision-based). This motivates us to resort to hybrid-based solutions to address these shortages. However, the accuracy and robustness of hybrid ones still need improvement due to a lack of consideration of structure constraints and global temporal redundancy. To solve these problems, we present PoseVR, a novel architecture that leverages the fusion of 2D vision and 3D sensor information. To compensate for the shortage of the single-source solution, we propose a dual-branch fusion structure to eliminate the redundancy of global temporal information by integrating the continuity of local temporal information. Motivated by the prior knowledge of human physiological structure and joint location, a novel coarse-to-fine endpoint space strategy is introduced to formulate the edge point of the body as prior information for accurately predicting full-body pose. Furthermore, a spatial loss function is employed for hierarchical prediction to achieve better accuracy. We qualitatively and quantitatively evaluate the proposed PoseVR. Experimental results show that PoseVR achieves state-of-the-art performance. Code is available at https://github.com/wwwpkol/PoseVR .