The Moving Picture Experts Group (MPEG) standardized Video-based Point Cloud Compression (V-PCC) is an emerging coding standard for 3D dynamic point clouds. V-PCC projects point clouds into geometry and attribute videos and uses Versatile Video Coding (VVC) as a video encoder to improve compression efficiency, but it also results in a huge coding complexity. To reduce the coding complexity, we propose a fast Coding Unit (CU) decision algorithm based on a Support Vector Machine (SVM) for VVC coding of geometric and attribute videos in V-PCC. First, we extract different features based on CU types, including texture feature, coding feature, inter feature, and geometric feature. Second, we trained discriminators for various sizes of CUs and selected different weight factor for each discriminator to achieve a trade-off between coding complexity and Rate-Distortion (RD) performance. The experimental results show that the proposed fast decision method reduces the complexity by 27.49% with Bjónteggard Delta Bit Rate (BDBR) of 1.23% and 1.36% compared to the anchor VVC-based V-PCC.