Semantic segmentation models face a critical dilemma: dense prediction approaches achieve high accuracy but produce visually disruptive fragmented masks, while generative methods create coherent masks but sacrifice spatial precision and incur prohibitive computational costs. Examining this trade-off, we find that generative models encode valuable shape priors that could transform semantic segmentation if efficiently transferred. Building on this insight, we present GUIDE (Generative Understanding Injection for Dense Estimation), a novel framework that injects generative coherence knowledge through a layer specific Soft-Guidance Query (SG-query) mechanism during training, then deploys this knowledge at inference time without the generative branch. The key innovations are a layer adaptive guidance attention bias that modulates cross-attention operations within each decoder layer, enhancing focus on regions conforming to learned shape priors while suppressing fragmentation, complemented by multi-scale Wasserstein distribution alignment that enables comprehensive shape-level knowledge transfer. Experiments across Cityscapes, ADE20K, and Hypersim datasets demonstrate that GUIDE reduces fragmentation by 59.4% while improving accuracy (+1.9% mIoU) with minimal computational overhead (4% parameter increase), achieving fragment-free segmentation without the substantial costs of existing generative approaches. This training-time learning, inference-time guidance paradigm efficiently combines the complementary strengths of both segmentation approaches.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

GUIDE: Generative Understanding Injection for Dense Estimation via Enhanced Shape Coherence

  • Shengye Yang,
  • Jie Liu,
  • Qingyang Zhang

摘要

Semantic segmentation models face a critical dilemma: dense prediction approaches achieve high accuracy but produce visually disruptive fragmented masks, while generative methods create coherent masks but sacrifice spatial precision and incur prohibitive computational costs. Examining this trade-off, we find that generative models encode valuable shape priors that could transform semantic segmentation if efficiently transferred. Building on this insight, we present GUIDE (Generative Understanding Injection for Dense Estimation), a novel framework that injects generative coherence knowledge through a layer specific Soft-Guidance Query (SG-query) mechanism during training, then deploys this knowledge at inference time without the generative branch. The key innovations are a layer adaptive guidance attention bias that modulates cross-attention operations within each decoder layer, enhancing focus on regions conforming to learned shape priors while suppressing fragmentation, complemented by multi-scale Wasserstein distribution alignment that enables comprehensive shape-level knowledge transfer. Experiments across Cityscapes, ADE20K, and Hypersim datasets demonstrate that GUIDE reduces fragmentation by 59.4% while improving accuracy (+1.9% mIoU) with minimal computational overhead (4% parameter increase), achieving fragment-free segmentation without the substantial costs of existing generative approaches. This training-time learning, inference-time guidance paradigm efficiently combines the complementary strengths of both segmentation approaches.