When capturing videos with cameras, noise can occur due to variations in lighting conditions, movements of subjects or cameras, and the quality of camera sensors. The presence of noise complicates object detection and tracking. To mitigate these issues, video denoising techniques have been developed. Numerous video denoising techniques have been proposed, with convolutional neural networks (CNNs) being predominant in recent years. While CNN-based methods achieve high accuracy in video denoising, they often lack causality because they utilize temporal information from the next frame as well as the current frame, which poses challenges for real-time processing without degrading image quality. Therefore, in this paper, we propose a new architecture that enhances processing speed by improving the Efficient Multi-stage Video Denoising (EMVD), which is of the state-of-the-art video denoising methods. Through experiments, it was demonstrated that the proposed method reduced the computation time by approximately 75% while limiting the accuracy degradation to 0.7% compared to conventional methods.