Cost-Effective Robot Arm Teleoperation Via Human Pose Tracking with Monocular Camera

Robot teleoperation is an essential tool for performing complex tasks that require dexterity beyond the capabilities of state-of-the-art algorithms. Existing teleoperation methods are often non-intuitive for human operators or require special sensors and equipment, making them cost-ineffective and impractical in many scenarios. In this paper, we propose a robot arm teleoperation framework that relies on monocular camera images. The proposed framework first uses lightweight neural networks for estimating the human operator body pose and recognizing their hand gesture. An efficient inverse kinematics algorithm then finds the desired robot arm configuration, achieving end-effector motion that imitates operator’s wrist movement. Our teleoperation framework can be executed on an average laptop using a web camera with any robotic arm that has a ROS interface. We verified its performance in real-world experiments with a Kinova Jaco robotic arm, demonstrating the ability to grasp and move an object in the environment.