Predicting pedestrian crossing intention is vital for enabling safe and anticipatory behaviour in autonomous vehicles (AVs), particularly in complex and visually degraded urban settings. Existing approaches often rely on high-quality visual inputs, dense semantic maps, and computationally intensive architectures, limiting their generalizability and real-time applicability. To address these limitations, we propose an end-to-end framework that unifies visual appearance, motion dynamics, and symbolic scene attributes for robust intention forecasting. Our method integrates a MobileViT-S backbone for visual encoding, a GRU-based trajectory encoder for motion, and a template-driven semantic context encoder whose outputs are embedded via MiniLM-L6-v2. These heterogeneous cues are fused through an Attention-Gated Cross-Modal Fusion Block that adaptively aligns modalities via cross-attention and gating mechanisms. Evaluated on the PIE and JAAD datasets, our model achieves competitive results (AUC: 0.92 on PIE, 0.89 on JAAD) using only ~0.99M trainable parameters.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Language Model-Guided Visual Reasoning for Scene-Aware Pedestrian Intention Prediction

  • Neha Sharma,
  • Chhavi Dhiman,
  • S. Indu

摘要

Predicting pedestrian crossing intention is vital for enabling safe and anticipatory behaviour in autonomous vehicles (AVs), particularly in complex and visually degraded urban settings. Existing approaches often rely on high-quality visual inputs, dense semantic maps, and computationally intensive architectures, limiting their generalizability and real-time applicability. To address these limitations, we propose an end-to-end framework that unifies visual appearance, motion dynamics, and symbolic scene attributes for robust intention forecasting. Our method integrates a MobileViT-S backbone for visual encoding, a GRU-based trajectory encoder for motion, and a template-driven semantic context encoder whose outputs are embedded via MiniLM-L6-v2. These heterogeneous cues are fused through an Attention-Gated Cross-Modal Fusion Block that adaptively aligns modalities via cross-attention and gating mechanisms. Evaluated on the PIE and JAAD datasets, our model achieves competitive results (AUC: 0.92 on PIE, 0.89 on JAAD) using only ~0.99M trainable parameters.