PointerNet with Local and Global Contexts for Natural Language Moment Localization

We consider the problem of natural language moment localization. Given an untrimmed video and a natural language query, we aim to automatically retrieve a semantically relevant moment in the video referred by the query sentence. Most existing methods work by projecting visual and linguistic data into feature embedding space, then matching the semantic similarity or ranking a set of pre-defined segments to select the moment. In this paper, we propose a novel PointerNet with local and global contexts to solve this problem. Our proposed model first uses a recurrent network over words to interact visual and linguistic features in a fine-grained fashion. The word recurrence represents each clip as a multimodal feature that captures the fine-grained interaction of each clip with all words in the query sentence. It then uses another bi-directional recurrent network that processes all clips in the video. The clip recurrence refines the local context information of each clip and produces a global context representation of the entire video. Finally, the global video context and the local context of each clip are jointly used to determine the start and the end positions of the moment. Extensive experimental results demonstrate the effectiveness of our proposed method.