Comparing LSTM and Transformer for Video Depth Estimation

Accurate depth estimation from monocular video is critical for robotics applications such as simultaneous localization and mapping (SLAM) and navigation. Monocular depth estimation from video can be improved by incorporating temporal information across frames. The recently introduced sequence modeling techniques of recurrent long-short-term memory (LSTM) networks and Transformer architectures provide two potential approaches for aggregating temporal cues. This work presents a comparative study of using LSTM and Transformer modules for video depth prediction. The proposed depth pipeline extracts optical flow features between frames and passes them to either an LSTM or Transformer encoder before decoding into a depth map prediction. Compared to LSTM, the Transformer’s ability to capture long-range dependencies allows it to propagate information more effectively across long sequences. It is shown that the Transformer outperforms LSTM models by five- to sixfold in depth map estimation based on standard metrics. This analysis provides insights into the advantages of Transformer over recurrent LSTM models for aggregation of temporal signals in depth estimation and other similar sequence prediction tasks. The Transformer’s ability in aggregating motion across sequences holds promise for more robust spatial perception.