Emerging applications such as virtual customer service, online education, and digital humans demand real-time generation of multimodal content with stringent requirements on throughput, latency, and reliability. Traditional solutions built on synchronous REST APIs or basic queue models exhibit tight coupling, limited scalability, and weak fault isolation, making them unsuitable for industrial-scale workloads. This paper presents a message-driven distributed framework for multimodal video generation and interaction that decouples task and result flows, supports a multi-producer/multi-consumer pattern, and achieves linear scalability and fault tolerance through asynchronous task orchestration. The pipeline integrates five AI modules in succession—large-language-model dialogue generation, text-to-speech synthesis, Wav2Lip lip synchronization, lightweight semantic motion matching, and MimicMotion-based motion refinement—which together yield lifelike video, even for low-resource languages. Four core roles constitute the system: task producers implemented in Go, which embed trace IDs and push JSON jobs to a video tasks queue; a message broker that handles routing, retries, and dead-letter queues; Python workers that execute the multimodal chain and publish finished videos to a video results exchange; and result listeners that fetch videos by routing key and return them to clients. We detail the architecture, module dependencies, and containerized deployment scripts, and we release complete open-source code with one-click reproduction tools, offering the research and industrial communities a highly reproducible, maintainable, and easily extensible engineering paradigm for multimodal content generation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Distributed Multimodal Video Generation and Interaction Framework Based on Message-Driven Middleware

  • Mamyr Altaibek,
  • Altanbek Zulkhazhav,
  • Gulmira Bekmanova,
  • Assel Omarbekova,
  • Banu Yergesh,
  • Dinara Kabdylova,
  • Beibarys Sultan,
  • Dilmukhan Andakov

摘要

Emerging applications such as virtual customer service, online education, and digital humans demand real-time generation of multimodal content with stringent requirements on throughput, latency, and reliability. Traditional solutions built on synchronous REST APIs or basic queue models exhibit tight coupling, limited scalability, and weak fault isolation, making them unsuitable for industrial-scale workloads. This paper presents a message-driven distributed framework for multimodal video generation and interaction that decouples task and result flows, supports a multi-producer/multi-consumer pattern, and achieves linear scalability and fault tolerance through asynchronous task orchestration. The pipeline integrates five AI modules in succession—large-language-model dialogue generation, text-to-speech synthesis, Wav2Lip lip synchronization, lightweight semantic motion matching, and MimicMotion-based motion refinement—which together yield lifelike video, even for low-resource languages. Four core roles constitute the system: task producers implemented in Go, which embed trace IDs and push JSON jobs to a video tasks queue; a message broker that handles routing, retries, and dead-letter queues; Python workers that execute the multimodal chain and publish finished videos to a video results exchange; and result listeners that fetch videos by routing key and return them to clients. We detail the architecture, module dependencies, and containerized deployment scripts, and we release complete open-source code with one-click reproduction tools, offering the research and industrial communities a highly reproducible, maintainable, and easily extensible engineering paradigm for multimodal content generation.