Procedure segmentation in videos with Bayesian Neural ODE model (BNODE)

Video event localization, the task of accurately identifying and localizing events within a video, poses a significant challenge due to the complex nature of video data. The abundance of information embedded in videos requires sophisticated models to effectively capture and process the relevant features for precise event localization. Traditional video event localization methods based on recurrent neural networks (RNNs) face challenges due to their limitations in handling long-range dependencies and sensitivity to initial network parameters. As an alternative, neural ordinary differential equations (neural ODEs) offer distinct advantages over RNNs. However, it is imperative to note that neural ODEs are inherently deterministic, rendering them less effective in modeling the inherent uncertainty in real-world videos. This paper proposes Bayesian-based neural ordinary differential equations (Bayesian neural ODEs—BNODEs) as an alternative approach for video event localization. BNODEs based on variational inference techniques are not scalable to large models, as they require complex optimization, instead, we propose Bayesian neural ODE based on Monte Carlo dropout. This method offers several advantages over both RNNs and standard neural ODEs, including the ability to capture extended context or temporal dependencies, being less sensitive to the initial weights of the network parameters, improved modeling of inherent uncertainty, and producing competitive results with less trainable parameters. The proposed encoder–decoder model powered by BNODEs achieves state-of-the-art results on the YouCook2 dataset, outperforming existing methods by a significant margin including those relying on convolution neural networks (CNNs) and bimodal inputs.