In this work, we investigate a cooperative formation control problem for a swarm of wheeled mobile robots (WMRs), by combining fuzzy logic systems (FLS) with critic-actor architecture reinforcement learning (RL). With the performance indicators of minimum energy consumption and minimum collaborative error, an observer of critic-actor RL is designed to estimate the unmeasurable states of each robot, while the critic and actor network is applied to auxiliary realize the cooperative optimal control performance for all the robots. Then, a distributed formation control strategy is proposed by consists of a kinematic controller and a dynamic torque controller. However, the underactuated and nonholonomic characteristics of WMRs governed by nonholonomic Lagrangian dynamics make the analytical solution of the Hamilton-Jacobi-Bellman (HJB) equation particularly challenging. To overcome this, the critic and actor update laws are derived from the negative gradient of a simple positive function, obtained from the partial derivative of the HJB equation, rather than from the squared residual of the approximated HJB formulation. Finally, theoretical analysis and simulation results validate the proposed method, demonstrating its effectiveness in achieving optimal formation tracking under uncertain dynamics.