Robust Self-contact Detection Based on Keypoint Condition and ControlNet-Based Augmentation

Existing self-contact detection methods have difficulty detecting dense per-vertex self-contact. Dataset collection for existing self-contact detection methods is costly and inefficient, as it requires different subjects to mimic the same pose. In this paper, we propose a generation-to-generalization approach by utilizing ControlNet to augment existing datasets. Based on that we develop a keypoint-conditioned neural network that can successfully infer per-vertex self-contact from a single image. With the extended dataset synthesized by ControlNet, our network requires only one real subject training data to achieve satisfactory individual generalization ability. Experiments verify the effectiveness of our proposed method and the improvement of the network’s generalization with synthetic data.