Scholarlink

<p>Markov Decision Process (MDP) serves as the fundamental mathematical framework for numerous sequential decision-making problems in real-world applications, with extensive implementation in complex Cyber-Physical Systems (CPS) such as autonomous driving and robotic control, where safety assurance is paramount. Building upon DRLFailureMonitor, our previous work on failure monitoring for Deep Reinforcement Learning methods, this paper presents FMMDP, a novel grey-box failure monitoring framework applicable to all MDP-based decision processes. FMMDP models failure evolution processes by capturing state-action sequences and employs multivariate time series classification techniques to learn failure patterns. Our framework introduces a state extraction module, enabling broad applicability across diverse MDP environments including both sensor-based and vision-based systems. Comprehensive evaluation across six distinct environments demonstrates FMMDP’s superior monitoring capabilities, achieving perfect recall (1.0) with an average false positive rate of merely 0.024, while providing an average advance warning of 19 timesteps before failures occur. Comparative analysis against ThirdEye and MC-Dropout reveals FMMDP’s significant advantages in both accuracy and computational efficiency. Parameter sensitivity studies indicate that state-action compositions offer optimal balance between precision and warning time, with sequence lengths of 20-30 timesteps yielding optimal performance across most environments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

FMMDP: failure monitoring approach for DNN-based Markov decision process

  • 期刊论文

  • 发表日期：

    2025年12月5日

查看全文
  • Yi Cai,
  • Weibin Lin,
  • Chao Jing,
  • Zhihao Liu,
  • Zheng Zheng

摘要

Markov Decision Process (MDP) serves as the fundamental mathematical framework for numerous sequential decision-making problems in real-world applications, with extensive implementation in complex Cyber-Physical Systems (CPS) such as autonomous driving and robotic control, where safety assurance is paramount. Building upon DRLFailureMonitor, our previous work on failure monitoring for Deep Reinforcement Learning methods, this paper presents FMMDP, a novel grey-box failure monitoring framework applicable to all MDP-based decision processes. FMMDP models failure evolution processes by capturing state-action sequences and employs multivariate time series classification techniques to learn failure patterns. Our framework introduces a state extraction module, enabling broad applicability across diverse MDP environments including both sensor-based and vision-based systems. Comprehensive evaluation across six distinct environments demonstrates FMMDP’s superior monitoring capabilities, achieving perfect recall (1.0) with an average false positive rate of merely 0.024, while providing an average advance warning of 19 timesteps before failures occur. Comparative analysis against ThirdEye and MC-Dropout reveals FMMDP’s significant advantages in both accuracy and computational efficiency. Parameter sensitivity studies indicate that state-action compositions offer optimal balance between precision and warning time, with sequence lengths of 20-30 timesteps yielding optimal performance across most environments.