Recently, we have witnessed the widespread adoption of Machine Learning (ML) techniques in the domain of network intrusion detection. However, most ML-based network intrusion detection systems (NIDSes) operate under the closed-world assumption where the training and test datasets consist of uniformly distributed normal or abnormal samples. This assumption often fails to hold in real-world scenarios, where both malware and normal behaviors evolve over time, leading to dynamic changes in network distribution (known as concept drift). In the paper, we design a deep reinforcement learning (RL) based network intrusion detection system capable of automatically adapting to concept drift in the open world. Particularly, we introduce a quality-driven drift detector to lightweight select and label drift samples. Then, our method can effectively adapt both normality and abnormality shifts via a sampling function within a dynamic simulation environment. We validate the effectiveness of our method using two public datasets exhibiting real-world drift. Experimental results demonstrate that our method achieves an F1 score of 0.93, outperforming the state-of-the-art(SOTA) methods.