In recent years a major breakthrough was made in the field of generative Artificial Intelligence. Although optimizing GPU usage is often emphasized during the training phase, challenges of efficient inference are equally critical, especially as models transition from research to real-world applications. In this article, strategies for optimizing and GPU usage during inference to minimize latency and maximize performance were evaluated. The test scenarios involved transferring parts of the model to CPU, removing most resource-consuming parts, reducing precision of weight’s bit representation, model quantization, and commands precompiling. The standard procedure contained the generation of graphics using the Stable Diffusion 3 model with simultaneous measurement of time and memory consumption. It was shown that, simple strategies can reduce VRAM usage by 70% and accelerate inference by 30%. The second phase of research referred to answering the question of how proposed methods impact the quality of generated results. Verification of results using the CLIP score showed that there is no significant deterioration in quality of final results.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Balancing the Load: Optimizing Inference of Generative Model on Example of Stable Diffusion

  • Łukasz Popek,
  • Piotr Bilski,
  • Rafał Perz

摘要

In recent years a major breakthrough was made in the field of generative Artificial Intelligence. Although optimizing GPU usage is often emphasized during the training phase, challenges of efficient inference are equally critical, especially as models transition from research to real-world applications. In this article, strategies for optimizing and GPU usage during inference to minimize latency and maximize performance were evaluated. The test scenarios involved transferring parts of the model to CPU, removing most resource-consuming parts, reducing precision of weight’s bit representation, model quantization, and commands precompiling. The standard procedure contained the generation of graphics using the Stable Diffusion 3 model with simultaneous measurement of time and memory consumption. It was shown that, simple strategies can reduce VRAM usage by 70% and accelerate inference by 30%. The second phase of research referred to answering the question of how proposed methods impact the quality of generated results. Verification of results using the CLIP score showed that there is no significant deterioration in quality of final results.