Fast QTMT Decision for H.266/VVC via Jointly Leveraging Neural Network and Machine Learning Models

The latest video coding standard H.266/VVC has significantly improved the compression efficiency compared to its predecessor H.265/HEVC. One of the key technologies in H.266/VVC is the QuadTree with nested Multi-type Tree (QTMT), which enhances the coding performance at the cost of increased time complexity. To accelerate the QTMT partitioning process while mitigating performance degradation, this paper models the Coding Unit (CU) partition as a hierarchical decision process and jointly leverages machine learning (ML) and neural network (NN) models for fast CU partition. Specifically, for large-size CUs, we devise an NN model to predict the probability of each candidate partition mode automatically; to further accelerate the above process, we additionally embed an ML algorithm before the NN model, for the fast determination of binary-tree horizontal (BTH) mode, a commonly-used mode in H.266/VVC. On the other hand, for small-size CUs, the neural model is inadequate due to the insufficient pixels for correlation exploration. We thus directly apply the ML model for early terminating the horizontal or vertical partition of each CU. We exemplify the proposed method using typical ML models such as Support Vector Machine (SVM) and Decision Tree (DT) and an NN model of 3.7 MB model size. Experimental results demonstrate that the proposed method achieves 45.04% encoding time reduction over the H.266/VVC reference software VTM-15.0 with only 1.01% BD-BR increase, which remarkably outperforms state-of-the-art works.