Mutually Complementary HAR System Using IMU-Based Wearable Devices and Computer Vision

Human Activity Recognition (HAR) is an effective approach to achieving intuitive Human-Computer Interaction (HCI). Pose estimation, which targets the entire body, is a major area of HAR, with a focus on skeleton estimation using computer vision. The primary advantage of estimating the skeleton using computer vision is the ability to reduce the dimensions of the data to be processed ultimately (advantage of skeleton extraction). However, there are physical limitations due to the nature of computer vision systems. Instances where the target is detected overlaps with other objects or the image is unclear due to ambient lighting serve as examples [1]. Furthermore, the skeleton estimation system represents the joints corresponding to the landmarks as 3-axis data. Given the nature of 2D data, the accuracy of the remaining axis is relatively low. To overcome these issues, we propose the application of wearable devices based on Inertial Measurement Units (IMUs) in conjunction with computer vision systems. IMUs are well-suited for measuring the movement of a target due to their lightweight and compact size. Although those are vulnerable to noise and require preprocessing [2], this drawback can be overcome with algorithms at the level of Kalman filters [3], and the required preprocessing level is lower when applied to AI (Artificial Intelligence). In this study, we propose a system that mutually complements IMUs and computer vision systems. The most widely used Pose estimation framework is Google's MediaPipe, which defines 32 landmarks [4]. However, wearing 32 IMUs is not cost-effective or convenient. For this reason, this study proposes wearing 10 IMU sensors on the left and right shoulders, hips, knees, ankles, and wrists. The contributions of the proposed hybrid system are as follows: In a 2D computer vision system, when the rest of one axis is incomplete, it can be complemented by IMU data. In the IMU system, when distance data between joints or position of joints is incomplete, it can be complemented by computer vision. When the target to be recognized is overlapped, IMU data can be used as ground truth. The performance of the proposed system is evaluated based on the skeleton extracted using both IMU data and images. A comparison and analysis are conduct-ed between cases where the skeleton is estimated based on the image and using IMU data.