Text-to-image diffusion models have become a cornerstone of modern generative AI, offering high-quality synthesis yet remaining constrained by their black-box nature, which limits controllability and interpretability. In this work, we propose a hybrid transformer-conditioned diffusion model that integrates UNet-based denoising with multi-head cross-attention transformer blocks at critical latent stages of the diffusion process. The architecture is trained on a curated set of 50,000 samples from DiffusionDB with a 200-step latent diffusion schedule. Text prompts are encoded using a 16-token BERT encoder and mapped into a 256-dimensional latent feature space. Cross-attention layers with eight heads are interlaced within the UNet bottleneck and decoder, enabling token-to-region correspondence and fine-grained semantic propagation. To ensure interpretability, we design an explainability framework that combines hierarchical token-level attention heat maps, temporal attention rollouts, and perceptual ablation studies based on learned image patch similarity. Analysis reveals that object tokens remain spatially and temporally consistent, while attribute tokens demonstrate sharper temporal volatility. Jensen–Shannon divergence quantifies this redistribution of attention across diffusion steps. Experimental evaluation against a standard UNet diffusion baseline demonstrates clear improvements: Fréchet Inception Distance decreases by 19.6, CLIP alignment score increases by 5.4, and Inception Score improves by 18.6. Moreover, attention coherence improves by 22%, underscoring the gains in explainability. The proposed framework establishes a pathway toward accountable, high-fidelity, and interpretable text-to-image synthesis. Beyond performance, it supports critical tasks such as bias evaluation, fairness auditing, and quality assurance, offering a robust foundation for the next generation of explainable generative AI systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Decision Flow Tracing and Word Impact Analysis in Hybrid Transformer-Conditioned Diffusion Models for Text-to-Image Generation

  • Aldrin P. Thomas,
  • Shiju George,
  • N. Anand Raj,
  • S. Mohemmed Ajaz,
  • Midhun Shaju,
  • V. Akil Nasim

摘要

Text-to-image diffusion models have become a cornerstone of modern generative AI, offering high-quality synthesis yet remaining constrained by their black-box nature, which limits controllability and interpretability. In this work, we propose a hybrid transformer-conditioned diffusion model that integrates UNet-based denoising with multi-head cross-attention transformer blocks at critical latent stages of the diffusion process. The architecture is trained on a curated set of 50,000 samples from DiffusionDB with a 200-step latent diffusion schedule. Text prompts are encoded using a 16-token BERT encoder and mapped into a 256-dimensional latent feature space. Cross-attention layers with eight heads are interlaced within the UNet bottleneck and decoder, enabling token-to-region correspondence and fine-grained semantic propagation. To ensure interpretability, we design an explainability framework that combines hierarchical token-level attention heat maps, temporal attention rollouts, and perceptual ablation studies based on learned image patch similarity. Analysis reveals that object tokens remain spatially and temporally consistent, while attribute tokens demonstrate sharper temporal volatility. Jensen–Shannon divergence quantifies this redistribution of attention across diffusion steps. Experimental evaluation against a standard UNet diffusion baseline demonstrates clear improvements: Fréchet Inception Distance decreases by 19.6, CLIP alignment score increases by 5.4, and Inception Score improves by 18.6. Moreover, attention coherence improves by 22%, underscoring the gains in explainability. The proposed framework establishes a pathway toward accountable, high-fidelity, and interpretable text-to-image synthesis. Beyond performance, it supports critical tasks such as bias evaluation, fairness auditing, and quality assurance, offering a robust foundation for the next generation of explainable generative AI systems.