Extended Temporal Convolutional Networks with Multi-scale Attention for Video Description
摘要
Recent advancements in deep convolutional networks have excelled in video description tasks. We present Extended Temporal Convolutional Networks (ETCN), a novel architecture designed for video description tasks. ETCN integrates spatial convolutional encoders with deep temporal modeling and multi-scale attention mechanisms in an end-to-end trainable framework. Unlike conventional models with fixed spatio-temporal receptive fields, ETCN is doubly deep—combining hierarchical spatial feature extraction with long-range temporal modeling. To enhance the accuracy and relevance of generated descriptions, we introduce an attention module into the LSTM decoder, allowing the model to dynamically focus on semantically important visual regions during each decoding step. Rather than encoding the entire input sequence into a fixed vector, the attention mechanism provides a context vector that guides word generation at every timestep, enabling fine-grained alignment between video content and textual output. This structure improves the model’s ability to capture rich temporal dynamics and semantic detail. Experimental results demonstrate that ETCN outperforms state-of-the-art methods in video-to-text generation by jointly optimizing visual representation and sequential word prediction.