Scholarlink

<p>Robust multimodal perception remains a challenge for autonomous vehicles and smart surveillance systems in general especially in situations of low light, occlusion, and environmental interference. Traditional multispectral detection pipelines typically suffer from problems of spectral misalignment between thermal and RGB inputs, low correlation between inter-modal features, as well as poor detection performance in congested or complex scenes, and low-visibility scenarios. To tackle these challenges, we present DyGraFT-CMCL (Dynamic Graph-based Fusion Transformer with Cross-Modal Contrastive Learning) as a single architecture for reliable detection of VRU detection in changing lighting and environmental conditions. We propose a model that fuses RGB and thermal modalities through a CrossFusion Transformer that incorporates spatial and channel attention to refine multi-scale feature representations. Additionally, we use a Cross-Modal Contrastive Learning (CMCL) approach to project RGB–thermal feature embeddings to a common latent space for spectral consistency and modality-invariance. Finally, we include a Dynamic Graph Relation Modeling (DGRM) module to learn object-to-object spatial relations for situational awareness in crowded urban traffic scenes. Furthermore, a mechanism for Dynamic Gating (DG) automatically modifies the contribution of individual modalities (RGB, depth, and thermal) based on the surrounding light, thereby adapting and ensuring maximum detection capability, day or night. Comprehensive tests on the KAIST Multispectral Pedestrian Dataset and the FLIR ADAS Thermal Dataset indicate the superior performance of DyGraFT-CMCL, which produced 98.4% Accuracy, 97.9% Precision, 97.5% Recall, 97.6% F1 score, 97.2% mean Average Precision and Average Miss Rate (MR) of 3.1%. The results demonstrate consistent performance across datasets, robustness and effectiveness of the proposed framework. DyGraFT-CMCL proposed in this work contributes a new baseline for multispectral object detection providing a scalable, real-time sensing system for safe and efficient autonomous driving, safe smart surveillance and safe intelligent transportation systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

DyGraFT-CMCL: a dynamic graph-based fusion transformer with cross-modal contrastive learning for robust multispectral vulnerable road users detection using thermal-visual image processing

  • 期刊论文

  • 发表日期：

    2026年7月2日

查看全文
  • K Arunkumar,
  • D. J. Ashpin Pabi,
  • R. Ragul Kannan,
  • Archana Ganesh Said

摘要

Robust multimodal perception remains a challenge for autonomous vehicles and smart surveillance systems in general especially in situations of low light, occlusion, and environmental interference. Traditional multispectral detection pipelines typically suffer from problems of spectral misalignment between thermal and RGB inputs, low correlation between inter-modal features, as well as poor detection performance in congested or complex scenes, and low-visibility scenarios. To tackle these challenges, we present DyGraFT-CMCL (Dynamic Graph-based Fusion Transformer with Cross-Modal Contrastive Learning) as a single architecture for reliable detection of VRU detection in changing lighting and environmental conditions. We propose a model that fuses RGB and thermal modalities through a CrossFusion Transformer that incorporates spatial and channel attention to refine multi-scale feature representations. Additionally, we use a Cross-Modal Contrastive Learning (CMCL) approach to project RGB–thermal feature embeddings to a common latent space for spectral consistency and modality-invariance. Finally, we include a Dynamic Graph Relation Modeling (DGRM) module to learn object-to-object spatial relations for situational awareness in crowded urban traffic scenes. Furthermore, a mechanism for Dynamic Gating (DG) automatically modifies the contribution of individual modalities (RGB, depth, and thermal) based on the surrounding light, thereby adapting and ensuring maximum detection capability, day or night. Comprehensive tests on the KAIST Multispectral Pedestrian Dataset and the FLIR ADAS Thermal Dataset indicate the superior performance of DyGraFT-CMCL, which produced 98.4% Accuracy, 97.9% Precision, 97.5% Recall, 97.6% F1 score, 97.2% mean Average Precision and Average Miss Rate (MR) of 3.1%. The results demonstrate consistent performance across datasets, robustness and effectiveness of the proposed framework. DyGraFT-CMCL proposed in this work contributes a new baseline for multispectral object detection providing a scalable, real-time sensing system for safe and efficient autonomous driving, safe smart surveillance and safe intelligent transportation systems.