<p>This paper is invested to address the model-free optimal formation control problem for multiple quadrotor aircrafts with a virtual leader. Each quadrotor is considered an underactuated system with six degrees of freedom and highly nonlinear dynamics. Moreover, it is worth mentioning that the communication between each quadrotor and the leader quadrotor is difficult to achieve. To address these problems, a distributed observer estimates the desired position of the quadrotors by using the data from the neighbors and itself. Then, a model-free optimal control scheme based on the Integral Reinforcement Learning (IRL) algorithm is applied for both position control and attitude control to achieve the desired formation. Thanks to the IRL algorithm and an Off-Policy approach, the optimal control policy can be learned by using the collected data of quadrotors without the knowledge of the dynamic system. A simulation result built in Matlab software is given to verify the effective formation flight of the proposed controller.</p>