Multiple input multiple output (MIMO) communication is widely considered in current Fifth-Generation (5G) networks for seamless data transmission, which solves the problems of biz size data traffic in other communication networks. One of the major challenges in MIMO networks is channel estimation (CE), which should be addressed for better data transmission. Even though many channel estimation techniques were suggested by researchers, the estimation accuracy is relatively low. To address these issues, the proposed research focuses on developing a hybrid optimization technique combining recurrent neural network-long short-term memory (RNN-LSTM) with improved particle swarm optimization (IPSO) and the Adam optimizer. Initially, the least squares (LS) method is used to estimate pilot blocks and collect channel responses for training the RNN-LSTM to achieve channel estimation (CE). The weight parameters of the network are optimized through a hybrid IPSO-Adam approach. The IPSO-RNN-LSTM model achieves a CE accuracy of 98.48%, a bit error rate (BER) of 20.62%, and a mean squared error (MSE) of 19.33%, outperforming the performance of the traditional techniques tunicate swarm optimization-based hyper convolutional neural network LSTM (TSO-hyper CNN-LSTM) and grasshopper electric fish optimization algorithm–bidirectional LSTM (G-EFOA-Bi-LSTM).