Self-supervised cross frame-rate event-frame depth learning without ground truth
摘要
Event-based cameras provide high-temporal-resolution motion information and wide dynamic range, making them suitable for depth estimation in challenging dynamic environments. However, existing depth learning approaches often rely on ground-truth supervision, handcrafted event representations, or computationally expensive fusion strategies, limiting scalability and real-time deployment. Recent self-supervised methods partially address annotation dependency but struggle with cross-frame-rate alignment, temporal instability, and inefficient fusion of event and frame modalities. To overcome these limitations, this study proposes a novel Self-Supervised Event–Frame Self-Attention Transformer (EF-SAT) framework for depth estimation without ground-truth depth supervision. The proposed method employs a Vision Transformer-based encoder to jointly model event streams and intensity frames, while a cross-frame-rate self-attention mechanism captures spatiotemporal dependencies across heterogeneous data rates. The framework further carries revolutionary self-sampling based primarily on bilinear interpolation, timescale-consistent intensity decoding, and ego-motion optimization that uses absolute trajectory error (ATE) and relative pose error (RPE) constraints to improve geometric balances in successive frames. Depth and pose are learned using photometric reconstruction, event–frame alignment, temporal consistency, and smoothness constraints in a fully self-supervised manner. Experiments conducted on the DAVIS-240 C event–frame dataset demonstrate that the proposed approach achieves superior performance, attaining an Absolute Relative Error of 0.114 and RMSE of 0.163, outperforming state-of-the-art deep learning baselines. Overall, the proposed EF-SAT model demonstrates robust, temporally stable, and accurate depth estimation in dynamic scenes, highlighting its potential for real-world applications such as robotics, autonomous navigation, and event-driven perception systems.