Multi-Stage Book Perception and Bimanual Manipulation for Rearranging Book Shelves

This paper presents an autonomous robotic system for rearranging books on a shelf. We combine a filtered multi-stage perception approach with a collision-aware manipulation strategy that takes advantage of the unique abilities of different end effectors. Our setup consists of a PR2 robot with a human-like robotic hand, but our ROS- and MoveIt-based software could readily generalize to other two-arm service robots with minor modifications. The system uses deep learning, contextual knowledge, and traditional computer vision methods to detect and recognize books for manipulation. The paper then discusses challenges and approaches for bimanual book manipulation, including the retrieval of a book between others and force-controlled motions for placement. Experiments demonstrate the approaches’ feasibility and indicate future challenges toward robust behavior. The code for the system is available from https://github.com/TAMS-Group/PBBM-robot-librarian .