A Cooperative Approach to Teleoperation Through Gestures for Multi-robot Systems

A teleoperation system uses motion interface approaches that allow sending commands remotely to a mobile robotic base. Traditional equipment uses specific components and, many times, stocks on the operator’s body, which may hinder their locomotion and teleoperation. In addition to being limited to teleoperation of only one robotic equipment, not being able to adapt to a multi-robot system. The multi-robot systems are generally used to increase the efficiency of the applications, having two or more robots working together to achieve a goal in a cooperative form, increasing the capacity and robustness of task accomplishments. In this sense, this paper presents the development of a cooperative approach to the teleoperation of multi-robot systems coupled with robot manipulators. This work uses the framework MediaPipe Hands to capture 2D points of the hand of the operator through RGB images combined with depth data through a Kinect sensor to get 3D points of the hand of the operator. These data are used to send positions that the mobile base and the robot manipulators must assume during the teleoperation. Experiments were conducted to demonstrate the algorithm’s efficiency and precision during the realization of object pick-up tests. A video of the experiment was made and can be accessed through the link: https://youtu.be/10cVwcjPYOE .