Diffusion-based Text-to-Video (T2V) models have achieved significant advancements in producing high-quality videos. Nevertheless, prior research frequently diverges from human aesthetic preferences, demonstrates unnatural motion, and lacks overall visual quality. Current methods for aligning human preferences in other modalities are not directly applicable to T2V, as they suffer from inaccurate reward signals, challenges in modeling value functions, and training instability. To tackle these challenges, we propose VidDPOK (Video Diffusion Policy Optimization with KL-regularization), a reinforcement learning-based fine-tuning method designed for pre-trained T2V models. VidDPOK features a multi-level differentiable reward model that integrates image-level and video-level feedback to effectively evaluate temporal consistency. Additionally, it utilizes a value function network for video-text pairs, enabling precise assessment of semantic and visual quality. Furthermore, KL divergence regularization constrains policy updates, thereby enhancing both semantic consistency and generative diversity while stabilizing the training process. Systematic evaluation using the VBench benchmark reveals that CogVideoX-2B, fine-tuned with VidDPOK, significantly outperforms baseline models in both automated metrics and human evaluations, even exceeding the performance of the larger CogVideoX-5B. These findings indicate that VidDPOK effectively enhances video generation quality and aligns outputs with human preferences, providing a generalizable solution for T2V models.