Imitation Learning of Long-Horizon Manipulation Tasks Through Temporal Sub-action Sequencing

This research proposes an approach to long-horizon manipulation which uses video and kinesthetic demonstrations to imitate human actions. The task learning process involves two stages. To learn the sequence of the sub-actions in the video demonstration, the Task Sequencing Network (TSNet) - a hybrid neural network made up of Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Connectionist Temporal Classification (CTC) loss, is used in the first stage. Through dynamic movement primitive (DMP) models, task-agnostic task primitives are learned in the second stage via kinesthetic demonstrations. To encode the semantic relationship between the sub-actions and the objects, a Multi-relational Embedding Network (MRE) using YOLOv4 for object detection is used to estimate the affordances associated with the objects in the scene. For tasks like liquid pouring, table cleaning and object placement, the proposed imitation learning approach learns task planning and execution in a decoupled manner, resulting in effective sub-action sequencing and quicker and more precise learning of sub-action execution.