MoCap-Video Data Retrieval with Deep Cross-Modal Learning

Cross-modal retrieval between video and motion capture (MoCap) data facilitates efficient reuse of human motion data in either skeletal or video format. For this purpose, we propose a deep cross-modal learning model for cross-modal retrieval between MoCap data and video data. First, we use a graph convolution-based network and a 3D convolution-based network to extract features from MoCap data and video data, respectively. In addition, we propose to use a pre-defined common subspace to maximize the inter-class variation and minimize the intra-class variation. Furthermore, we employ a similarity matrix to achieve the alignment between these two modalities and exploit their underlying correlations. For the purpose of experimental evaluation, due to the small amount of video data corresponding to the MoCap data in the public HDM05 dataset, we recorded a video dataset corresponding to the HDM05 motion capture dataset and performed cross-modal retrieval on it. The experimental results proved the effectiveness of the proposed scheme.