Human Joint Localization Method for Virtual Reality Based on Multi-device Data Fusion

Virtual reality (VR) utilizes computer vision, artificial intelligence and other techniques to enable interaction between users and virtual environments. In order to solve the problems of human joint localization based on single device, multi-device data fusion technology has been adopted. In this paper, a multi-device data fusion method is proposed based on HTC Vive and Kinect. Firstly, two devices are utilized to separately capture motion data of human joints and the two sets of data are aligned temporally and unified in coordinates. Then the weights are respectively assigned to the two sets of data based on the different location of the human body. Next, particle filtering is adopted to combine the two sets of data. Finally, a bidirectional long short-term memory (Bi-LSTM) neural network model is deployed, where the bone length loss is incorporated into the loss function to further improve the localization accuracy. Experiment results show that the localization accuracy of the proposed multi-device data fusion-based localization method outperforms that of the single device method.