With the widespread deployment and application of unmanned aerial vehicles (UAVs), a large amount of traffic is generated in highly dynamic UAV systems, along with numerous malicious attacks such as identity impersonation, denial of service, and GPS spoofing, which can cause significant security threats. Existing intrusion detection schemes suffer from slow detection speed or low detection accuracy in highly dynamic UAV systems. In this paper, we propose an intrusion detection scheme for UAV systems with deep reinforcement learning to accurately detect malicious attacks and optimize the detection policy (i.e., the selection of classifiers and their parameters) based on the feature information from UAV traffic. By applying a pre-learning method, this scheme uses a pre-trained model to obtain parameters of the evaluated network to avoid random unnecessary exploration and accelerate learning. This scheme formulates a punishment function using the F1 score, precision, accuracy, recall, latency, miss detection rate, and false alarm rate as metrics to evaluate the immediate risk of each detection policy and thus avoids risky detection policies that result in degradation of intrusion detection performance or even failure. Performance evaluation based on the CICIDS-2017 dataset shows that our proposed scheme outperforms baseline and classical classifiers with higher detection performance.