Spike-EFI: Spiking Neural Network for Event-Based Video Frame Interpolation

In order to model complex motions between video frames more accurately, many video frame interpolation methods have introduced event cameras to obtain additional high speed motion information. These methods use deep artificial neural networks (ANN) to process RGB images and event streams. However, traditional ANN-based methods are unable to effectively utilize the sparse and asynchronous nature of event streams, leading to unnecessary computations and increased energy consumption. As an alternative to ANN, spiking neural networks (SNN) can naturally process asynchronous and sparse event streams, reduce computational complexity and achieve lower energy consumption when combined with neuromorphic hardware. In this paper, we propose Spike-EFI, a lightweight fully spiking neural network for event-based video frame interpolation task. A spiking neural network with Leaky-Integrate and Fire (LIF) neuron is utilized to learn from a sensor fusion of RGB frames and event streams. This is also the first attempt to achieve video frame interpolation with neuromorphic computing paradigm. We trained and evaluated our network on public dataset, and the experimental results demonstrated that the proposed method has comparable performance to prior ANN-based methods. Benefitting from the event triggered computing paradigm of SNN, our method also achieves lower computational power consumption compared to ANN-based method.