Graph Foundation Models have emerged as powerful tools for transferring knowledge across diverse graph domains and tasks. However, we identify a critical limitation in current graph-text contrastive learning approaches: generic token dominance. Our analysis reveals that over 70% of attention weights concentrate on non-discriminative generic terms like “paper” and “research”, creating a semantic bottleneck that impairs cross-domain transfer capabilities. This phenomenon arises from optimization shortcuts in contrastive learning, where models favor high-frequency generic tokens over sparse discriminative features. To address this challenge, we propose GraphRefiner, a training-free calibration framework that enhances discriminative feature utilization without requiring model retraining. GraphRefiner employs token decomposition to identify and suppress generic terms, adaptive semantic modulation to preserve essential context, and discriminative similarity reasoning for fine-grained matching. Extensive experiments demonstrate that GraphRefiner achieves an average accuracy gain of 8.3% on cross-domain zero-shot tasks and 6.7% on few-shot transfer scenarios across standard benchmarks. Our findings highlight the importance of addressing optimization shortcuts in foundation models and provide a practical solution applicable to existing pretrained models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Mitigating Generic Token Dominance in Cross-Domain Foundation Model for Text-Attributed Graphs

  • Heng Zhang,
  • Haochen You,
  • Zijian Zhang,
  • Lubin Gan,
  • Hao Zhang,
  • Wenjun Huang,
  • Jin Huang

摘要

Graph Foundation Models have emerged as powerful tools for transferring knowledge across diverse graph domains and tasks. However, we identify a critical limitation in current graph-text contrastive learning approaches: generic token dominance. Our analysis reveals that over 70% of attention weights concentrate on non-discriminative generic terms like “paper” and “research”, creating a semantic bottleneck that impairs cross-domain transfer capabilities. This phenomenon arises from optimization shortcuts in contrastive learning, where models favor high-frequency generic tokens over sparse discriminative features. To address this challenge, we propose GraphRefiner, a training-free calibration framework that enhances discriminative feature utilization without requiring model retraining. GraphRefiner employs token decomposition to identify and suppress generic terms, adaptive semantic modulation to preserve essential context, and discriminative similarity reasoning for fine-grained matching. Extensive experiments demonstrate that GraphRefiner achieves an average accuracy gain of 8.3% on cross-domain zero-shot tasks and 6.7% on few-shot transfer scenarios across standard benchmarks. Our findings highlight the importance of addressing optimization shortcuts in foundation models and provide a practical solution applicable to existing pretrained models.