Dual Vision-Based Reinforcement Learning: Solving Robot Manipulation Task with Both Static-View and Active-View Cameras

In Deep Reinforcement Learning (DRL) practice related to robot manipulation, an open-source package called panda-gym provides a convenient approach to test different Reinforcement Learning (RL) algorithms by multiple predefined tasks. This package uses explicit data like coordinates as the observations instead of implicit data like images. To explore more on the feasibility of implementing a vision-based RL environment, we added two visual sources into the original panda-gym scenario. One source is named as active-view, which comes from the camera mounted on the robot. The other is called static-view, which comes from the stationary camera in front of the robot. Then we reformed the observations from previous scalar data to multidimensional data composed of visual data and positional data. To extract features from these mixed observations, Convolutional Neural Networks (CNN) were applied for extracting visual features, and Fully Connected Networks (FCN) were used for mapping actions. To foster the open-source platform, we performed the training on Stable-Baselines3, an open-source package with a set of reliable implementations of RL algorithms. In our experiment of a robot’s reaching a random tabletop object, we found the agent with dual view can overcome the object disappearance in active-view while using static-view to learn the optimal policy. The contrast is explicit when applying algorithms like DDPG after around 10,000 timesteps. It strongly emphasizes the importance of adding multi-angle views in the context of training a robot manipulation RL agent.