Recurrent neural networks (RNNs) are mostly useful in sequence modeling such as speech recognition, natural language processing, and time series analysis. In this survey, recurrent neural networks architecture, applications and advancements have been discussed. The importance of RNN in the classification and prediction has been discussed. RNN architecture and types of RNN have been explored. Application of RNN in processing sequential data, text, audio, and video are also explored. RNN has been successfully utilized in solving electrical engineering challenges such as smart grid optimization, fault detection, and predictive maintenance. This paper highlights these applications while discussing advancements and future prospects. In future, RNNs will be very helpful in the field of artificial intelligence.