This paper is invested to address the optimal tracking problem for the nonlinear quadrotor with unknown dynamics and external disturbance. First, the quadrotor is an underactuated system with six degrees of freedom and four inputs. Moreover, the quadrotor is considered under the influence of the parameter uncertainties due to the unknown payloads and the external disturbance. To address this problem, a robust control structure is given, which consists of the position and attitude controllers, the position controller achieves the position tracking and generates the attitude reference, and the attitude controller archives the stability of the quadrotor. By using the RL method, the optimal control policies can be learned without the dynamic information. Finally, a simulation built in Matlab software is given to verify the effectiveness of the proposed robust controller.