High-efficiency video coding (HEVC) standard is a widely used video coding standard in the consumer market due to its bit rate reduction capability and good video reconstruction. The visual appearance of the video sequences is improved using the in-loop filtering unit of the encoder. This in-loop filtering is proficient to handle the various real-time noise effectively. However, the encoder performance is deeply affected due to video sequences with bad weather conditions parameters such as rain, snow, and Haze. Conventional in-loop filtering module faces more issues due to the presence of statistical rain pattern in the video frames which directly affect the Rate-Distortion performance of the encoder. To address the above in-loop filtering-based issues, we have proposed dual stage deep rain streaks removal CNN-based in-loop filtering module. This in-loop filtering module consists of a modified Residual Dense block and Six stage scale feature aggregation module to study the various rain streaks during the training stage and eliminates the rain streaks from the successive video frames. The performance of the proposed dual-stage CNN-based in-loop filtering is analyzed on four classes of video sequences. The quantization unit of the HEVC encoder additionally introduces complexity to the rain removal process. To study the Rate–distortion characteristics of video encoder with proposed in-loop filtering, we have considered four Quantization Parameter(QP) values. The performance analysis was conducted on various rain streaks affected video sequences and the proposed HEVC encoder with rain streaks removal in-loop filtering provides a better PSNR value and higher bit rate reduction in RA and LDB configuration.