Physics-informed neural network (PINN) is a new paradigm for solving partial differential equations (PDEs). It is a data-driven approach, where layered neural networks approximate the solution of the PDE of interest via regularization to satisfy the PDE conditions. The PINN employs multilayer perceptrons (MLPs) in conjunction with numerical methods, e.g. Runge-Kutta (RK) to model discrete-time PDEs. However, the present networks’ structure exhibits constraints inherent in conventional numerical discretization approaches, particularly in handling iterative time-stepping processes. This is a critical challenge, which arises from an architectural standpoint. In addition, the PINN requires a dual-optimization approach to attain convergence, which can be computationally expensive. In this work, we propose physics-informed antisymmetric recurrent neural network (PIARNN), which encodes RK intermediate stages in the hidden state of the recurrent neural networks, thus enhancing the capabilities of the architecture of the PINN in learning complex nonlinear PDEs. We parameterize the recurrent connections with a strictly upper-triangular parameter vector, which achieves more robust and stable learning than the standard PINN. We conducted the experiments on Allen-Cahn and Burger’s equations, where the proposed framework achieved superior prediction performance across various RK stages while maintaining fewer learnable parameters. It was observed that the PIARNN can achieve a better result in approximating the solution of the PDE with a single optimizer than the PINN.