Egocentric Perception for Open Vocabulary Object Rearrangement

With the increasing need for service robots in households, it is essential to develop systems that can function in open environments and perform tasks such as organizing spaces. The diverse and ever-changing set of objects, home arrangements, and personal preferences make tidying household spaces a challenging problem. While several formulations of object rearrangement have been proposed as embodied AI tasks, they are limited to recognizable objects and universal arrangement preferences. We propose a method that performs tidying tasks in a personalized manner with open-vocabulary generalization to unseen objects. By leveraging geometric features learned through viewpoint invariance, inspired by CSR, and visual features obtained using a frozen CLIP model, we can learn tidying preferences. Our method consists of a perception module to recreate a map of the environment and a ranking module to predict rearrangement sequences to tidy the environment. Both modules generalize to an open set of objects and are learned from environment observations. Our perception module achieves a 27.1% accuracy in predicting correct object locations, and our ranking module correctly identifies desired receptacles for 49.0% of objects.