<p>Controllable video generation performs prompt-based editing of the input videos. Recent advances utilizing diffusion models have significantly improved the quality of generating/editing the videos. Nevertheless, achieving temporally consistent results, especially for long videos, remains an appealing yet challenging problem. In this paper, we propose a novel video diffusion model, namely GuidedVDM, which generates long videos (e.g., hundreds of frames) with strong temporal consistency by incorporating the guidance from the input videos. Technically, GuidedVDM capitalizes on an off-the-shelf controllable text-to-image model (e.g., ControlNet) conditioning on the input video and prompt to synthesize a reference image as appearance guidance. This guidance then serves as a control signal to video diffusion model to first generate sparse “Intra-coded frames” (I-frames). Next, GuidedVDM extracts optical flow of input video as motion guidance to warp the ready-made neighboring I-frames and produce dense “Bidirectional predictive frames” (B-frames) between I-frames. By reusing certain pixels from neighboring I-frames, GuidedVDM improves long-term temporal consistency across frames and accelerates controllable video generation. Extensive experiments demonstrate that GuidedVDM outperforms the state-of-the-art video diffusion model of TokenFlow, in temporal consistency, and exhibits <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(10\times \)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>10</mn> <mo>×</mo> </mrow> </math></EquationSource> </InlineEquation> faster generation speed. More remarkably, GuidedVDM can generate coherent multi-shot videos via synchronizing appearance guidance across shots.</p>