Spatial Temporal Diffusion Transformer (ST-DiT) has emerged as a new trend in generative diffusion models on text-to-video generation. As ST-DiT demonstrates great capabilities in practical scenarios, the demand for its inference increases significantly. However, due to the lack of performance studies and its distinct features, such as multi-dimensional attention mechanism, compared to transformer-based language model, our understanding of its inference workload is still limited. This paper aims to conduct an in-depth performance analysis on ST-DiT inference workloads, and provides comprehensive investigation on the identified performance problems, such as diverse performance constraints and varying memory footprint. We propose a resource-aware performance modeling method for ST-DiT and summarize its performance characteristics. Finally, this work conducts extensive experiments to validate our analysis and delivers valuable insights for efficient ST-DiT model deployment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Understanding the Inference Performance of Spatial Temporal Diffusion Transformer

  • Yu Li,
  • Yuanxin Wei,
  • Jiangsu Du,
  • Dan Huang,
  • Nong Xiao

摘要

Spatial Temporal Diffusion Transformer (ST-DiT) has emerged as a new trend in generative diffusion models on text-to-video generation. As ST-DiT demonstrates great capabilities in practical scenarios, the demand for its inference increases significantly. However, due to the lack of performance studies and its distinct features, such as multi-dimensional attention mechanism, compared to transformer-based language model, our understanding of its inference workload is still limited. This paper aims to conduct an in-depth performance analysis on ST-DiT inference workloads, and provides comprehensive investigation on the identified performance problems, such as diverse performance constraints and varying memory footprint. We propose a resource-aware performance modeling method for ST-DiT and summarize its performance characteristics. Finally, this work conducts extensive experiments to validate our analysis and delivers valuable insights for efficient ST-DiT model deployment.