Scholarlink

An AI pipeline to generate cinematic videos automatically from audio recordings. The pipeline begins with Whisper, which accurately transcribes audio into text. The transcript is then scanned with spaCy to identify sensible prompts and scene descriptions. The prompts are then employed to guide Stable Diffusion XL in producing high-definition, rich images that visually narrate the story. The model showed strong performance with a 4.6% Word Error Rate, FID score of 27.3. and 95.2.% subtitle alignment accuracy. The effectiveness of the pipeline was also confirmed through a user study, wherein average ratings of 4.5 for visual quality, 4.7 for readability of the subtitles, and 4.3 for narrative understanding suggested the promise of the pipeline for automated, engaging multimedia content generation.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Automated Cinematic Video Generation from Voice Recording Using Whisper, Stable Diffusion XL and SpaCy

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • H. Subrahmanya,
  • Rakshita Bolashetti,
  • Daneshwari Malladad,
  • Bipan Kumar,
  • Sharada K Shiragudikar

摘要

An AI pipeline to generate cinematic videos automatically from audio recordings. The pipeline begins with Whisper, which accurately transcribes audio into text. The transcript is then scanned with spaCy to identify sensible prompts and scene descriptions. The prompts are then employed to guide Stable Diffusion XL in producing high-definition, rich images that visually narrate the story. The model showed strong performance with a 4.6% Word Error Rate, FID score of 27.3. and 95.2.% subtitle alignment accuracy. The effectiveness of the pipeline was also confirmed through a user study, wherein average ratings of 4.5 for visual quality, 4.7 for readability of the subtitles, and 4.3 for narrative understanding suggested the promise of the pipeline for automated, engaging multimedia content generation.