The new generation video coding standard, H.266/VVC, introduces the quad-tree nested multi-type (QTMT) block partitioning structure and multiple inter coding modes, significantly improving coding efficiency but also increasing encoding time. To address this, we propose a coarse-to-fine fast partition decision (FPD) algorithm that collects both temporal and spatial information for inter CU partitioning. FPD first leverages co-located similarity between the current CU and its counterpart in the reference frame to capture global motion. High similarity indicates static regions, allowing early pruning of partition candidates. For CUs with low similarity, indicating complex local motions, we introduce a machine learning-based approach. Specifically, we extract temporal optical flow and spatial features (e.g., edges and gradients) to train a LightGBM classifier to predict the partition direction and skip the horizontal/vertical directions in advance. Experiments conducted under the common test condition of H.266/VVC demonstrate that our proposed FPD achieves a 37% runtime saving with only 0.99% coding performance loss, significantly surpassing state-of-the-art methods.