<p>Detecting and tracking the horizon line in maritime videos is crucial for various applications, such as navigation and situational awareness. Many existing algorithms for horizon line detection are highly accurate and perform well in maritime environments. However, most of these algorithms require substantial computational resources, making real-time processing impractical, particularly for large video frames. This paper presents a fast horizon line tracking algorithm that changes the region-of-interest (ROI) with an on-off control approach in each new frame. To enhance tracking accuracy, a new error metric is introduced and its Z-score is calculated to determine whether to use the full frame or ROI. The proposed method demonstrates significant improvement in processing speed while preserving accuracy. Specifically, in the Singapore On-board dataset videos, the proposed method doubled the processing speed while achieving a median positional and angular error of 1.03 pixels and 0.15 degrees, respectively. In the more challenging Buoy dataset, the total processing speed is increased by 27.1% compared to full-frame processing, with a low median positional and angular error of 0.99 pixels and 0.71 degrees, respectively. These results underscore the effectiveness of the adaptive ROI-based approach in achieving fast and accurate horizon line tracking in maritime environments. Additionally, this tracking method can be integrated with any horizon line detection algorithm, enhancing its applicability across different maritime and non-maritime environments. The result videos, showcasing the outcomes of this research, are available at <a href="https://doi.org/10.5281/zenodo.15369479">https://doi.org/10.5281/zenodo.15369479</a>.</p>