Considering the inherent complementary of multimodal data, the utilization of such data in gesture authentication has emerged as a significant domain within the biometric authentication landscape. However, the challenge of effectively integrating the highly sparse event stream data captured by event cameras with RGB imagery for gesture authentication remains an open question. This paper introduces an efficacious framework for gesture authentication that leverages both RGB imagery and event stream data. We have incorporated a transformer-based architecture specifically designed to model the event stream data and propose a fusion strategy for heterogeneous data with heterogeneous neural networks. Through rigorous experimentation conducted on our meticulously constructed dataset, the efficacy of the proposed framework has been empirically validated.