Respiratory rate (RR) is a critical physiological parameter for assessing respiratory function and is widely used in clinical applications. Non-contact monitoring of the RR enables continuous monitoring, avoiding the discomfort often associated with traditional contact methods, making it highly desirable in clinical applications. The existing method [1] could measure the RR from RGB videos by combining the optical flow method and the negative feedback crossover point method. The selection of feature points in the optical flow method significantly affects the accuracy of respiratory signals. To enhance the dynamic tracking performance of feature points, we propose to simplify the feature point evaluation criteria. Additionally, we utilize the temporal information of the feature points to adaptively optimize the feature point space. The crossover point method is prone to erroneous crossover points, while the negative feedback crossover point method can remove erroneous points but may also eliminate valid ones. Therefore, we further optimize the crossover point removal strategy based on the physiological information of the crossover points. In the Large-scale Bedside Respiration Dataset for Intensive Care (LBRD-IC), our proposal method estimates the RR with a mean absolute error (MAE) of 4.034 beats per minute (BPM) and a mean squared error (MSE) of 30.34 \(\text {BPM}^2\) , while the existing method achieves a MAE of 4.464 BPM and a MSE of 36.75 \(\text {BPM}^2\) . This demonstrates the effectiveness of our proposal method in clinical applications. The codes are publicly available at https://github.com/ZinChou/RR.