Due to the influence of both internal and external factors, fixed installed surveillance cameras often suffer from shakiness. In dynamic and complex scenes, frequent discontinuous depth variations and large foreground moving objects lead to multi-plane motion. This can lead video stabilization algorithms to misjudge local plane motion as global camera shakiness, resulting in stabilization failure or degraded performance. To address this problem, we propose a video stabilization algorithm based on the MeshFlow motion model. First, we propose a shakiness detection method and rules, which enables the stabilization algorithm to process only shaky frames, thus improving computational efficiency. Then, during motion estimation, we divide each frame into multiple mesh and construct a sparse motion field using motion vectors from mesh vertices to extract the camera’s shakiness trajectory. Finally, we apply Kalman filtering for trajectory smoothing, and use motion compensation to generate stabilized video. Experimental results show that the stabilized video achieves a PSNR improvement of 30% over the original video, only 0.23 dB lower than the SOFT algorithm. Additionally, the processing speed reaches 32 frames per second, which is 78% faster than SOFT algorithm, thereby meeting the requirements of practical applications.