<p>Video prediction which aims to generate future frames by learning from historical information is a challenging task due to dynamic motions and appearance changes in videos. Many existing methods are based on convolutional long short-term memory (ConvLSTM) networks or convolutional gated recurrent unit (ConvGRU) networks. PredRNN proposed recently has achieved promising results. However, the basic predictive unit of PredRNN is based on ConvLSTM, while ConvGRU which has similar performance using fewer parameters is not considered. To address this issue, we propose a novel Spatiotemporal Gated Recurrent Unit (ST-GRU) by introducing a spatiotemporal memory to ConvGRU and updating this memory using a gating mechanism. Using ST-GRU as the basic memory cell, we develop a predictive model where a spatiotemporal memory flow is applied to preserve the visual dynamics of different layers. In this case, the proposed model can extract and memorize spatial and temporal representation simultaneously. Experiments performed on several commonly used datasets demonstrate the effectiveness of the proposed model.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ST-GRU: spatiotemporal gated recurrent unit for video prediction

  • Jing Dong,
  • Junzhuo Zhang,
  • Ben Xie,
  • Jie Zhang,
  • Chang Liu,
  • Wei Cheng

摘要

Video prediction which aims to generate future frames by learning from historical information is a challenging task due to dynamic motions and appearance changes in videos. Many existing methods are based on convolutional long short-term memory (ConvLSTM) networks or convolutional gated recurrent unit (ConvGRU) networks. PredRNN proposed recently has achieved promising results. However, the basic predictive unit of PredRNN is based on ConvLSTM, while ConvGRU which has similar performance using fewer parameters is not considered. To address this issue, we propose a novel Spatiotemporal Gated Recurrent Unit (ST-GRU) by introducing a spatiotemporal memory to ConvGRU and updating this memory using a gating mechanism. Using ST-GRU as the basic memory cell, we develop a predictive model where a spatiotemporal memory flow is applied to preserve the visual dynamics of different layers. In this case, the proposed model can extract and memorize spatial and temporal representation simultaneously. Experiments performed on several commonly used datasets demonstrate the effectiveness of the proposed model.