Scholarlink

In complex urban environments, autonomous driving systems face persistent challenges due to unpredictable behaviors such as jaywalking and sudden lane changes, necessitating integrated approaches capable of accurate trajectory prediction and interpretable risk reasoning. This study proposes a multimodal framework that jointly performs trajectory prediction and explainable risk-aware decision reasoning by combining dynamic states of objects and contextual visual information extracted from vehicle-mounted imagery. The proposed framework comprises four modules: object behavior feature extraction, scene context feature extraction, driving scene context-augmented trajectory prediction, and risk-aware decision reasoning. Specifically, the model integrates behavioral features modeled through long-short term memory (LSTM)-based gated units and transformer encoders with visual context features to enhance trajectory forecasting. Furthermore, the DeepSeek-based multimodal large language model (MLLM) generates interpretable explanations by identifying critical risk objects, describing potential risks, and recommending vehicle actions. In our experiment, we confirm the feasibility and applicability of the proposed framework by implementing and applying it to Rank2Tell open dataset, achieving significant performance improvements (ADE: 10.972, FDE: 13.701, RMSE: 8.782) compared to baseline models, and qualitative comparisons across multiple language models, including DeepSeek (ours), LLaMA, Mistral based model, demonstrated superior reasoning capability and interpretability of the proposed multimodal approach.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Driving Scene Context-Augmented Trajectory Prediction with Risk-Aware Decision Reasoning Using Multimodal LLM

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Sunghun Kim,
  • Seokjun Hong,
  • Joobin Jin,
  • Byeongjoon Noh

摘要

In complex urban environments, autonomous driving systems face persistent challenges due to unpredictable behaviors such as jaywalking and sudden lane changes, necessitating integrated approaches capable of accurate trajectory prediction and interpretable risk reasoning. This study proposes a multimodal framework that jointly performs trajectory prediction and explainable risk-aware decision reasoning by combining dynamic states of objects and contextual visual information extracted from vehicle-mounted imagery. The proposed framework comprises four modules: object behavior feature extraction, scene context feature extraction, driving scene context-augmented trajectory prediction, and risk-aware decision reasoning. Specifically, the model integrates behavioral features modeled through long-short term memory (LSTM)-based gated units and transformer encoders with visual context features to enhance trajectory forecasting. Furthermore, the DeepSeek-based multimodal large language model (MLLM) generates interpretable explanations by identifying critical risk objects, describing potential risks, and recommending vehicle actions. In our experiment, we confirm the feasibility and applicability of the proposed framework by implementing and applying it to Rank2Tell open dataset, achieving significant performance improvements (ADE: 10.972, FDE: 13.701, RMSE: 8.782) compared to baseline models, and qualitative comparisons across multiple language models, including DeepSeek (ours), LLaMA, Mistral based model, demonstrated superior reasoning capability and interpretability of the proposed multimodal approach.