Soft Robot Learning with Vision

Vision-based deformable perception provides a novel solution to robotic perception using visual information to infer the interaction physics at the surface of contact. In this paper, we leverage the omni-adaptive capability of a soft network structure with differential stiffness by adding a monocular vision sensor at its bottom to track its spatial deformation while interacting with objects visually. We modeled the physical interaction of this soft finger and measured the distribution of the stiffness and friction coefficient through experiments to calibrate the parameters. Then, we use the vision sensor to collect data regarding the soft network’s deformation when interacted with a different contact force and position. Using a neural network modified from the AlexNet, we achieve a preliminary prediction model of the contact force and position using visual images of the soft network’s internal structure while interacting with probes. Our results show that the resulting system can achieve 90.00% accuracy in position prediction and 3.4% in the normalized root mean squared error in force prediction, which performs among state-of-the-art lab experiments. Future work involves integrating this soft vision finger in a field test with more object variation while installed on a gripper or used as a joystick for remote fine-motor control using visual servoing.