Segmentation for Grasping: A Holistic Framework for Autonomous Table Clearing

Autonomous clearing of food trays and crockery at hawker centres involve robotic tasks such as item recognition, grasp point estimation and grasp execution. Recent work treat the whole pipeline as one black box and try to predict grasp poses directly from the RBGD input. Although direct and end-to-end trainable, these models merely output highest ranked grasp poses of any reachable object, without any semantic concept of the item being picked up. This lack of scene understanding would eventually inhibit the optimization of the grasping algorithms as there is no way to willfully select a particular object for manipulation. In this work, we break down the grasp pose determination into several components and focus on solving them individually. We first parse the input scene with a convolutional neural network trained for instance segmentation, then compute a point cloud for each object of interest detected in the scene. The point cloud is registered to the library and the pose transformation is used to re-compute the grasp pose in the robot frame. The feasibility of the grasp pose and its trajectory as well as the efficiency of table clearing is considered in selecting which object to grasp first. Overall, we propose a holistic framework to clear diningware from the table autonomously by taking just a single view of the scene.