Robotic systems are increasingly deployed across diverse domains, performing multiple operations while continuously interacting with the environment and making decisions. This results in large volumes of multimodal data from various sources, including sensor readings, video feeds, and communication data. In this domain, process mining holds the potential for extracting behavioral patterns, identifying anomalies, and evaluating performance metrics in robotic systems. However, its application remains largely unexplored due to the fine-grained, multimodal nature of robotic data. In this work, we investigate how robotic data should be processed to enable process mining applications. We explore activity recognition techniques to transition from robotic fine-grained data to high-level activities, applying Conditional Random Fields to sensor data and fine-tuning the Florence-2 model for video data. Furthermore, we outline key challenges and opportunities in preparing robotic data for process mining, laying the groundwork for future research on process mining-based analysis of robotic systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enabling Process Mining on Multimodal Robotic Data

  • Flavio Corradini,
  • Sara Pettinari,
  • Barbara Re,
  • Lorenzo Rossi,
  • Massimiliano Sampaolo

摘要

Robotic systems are increasingly deployed across diverse domains, performing multiple operations while continuously interacting with the environment and making decisions. This results in large volumes of multimodal data from various sources, including sensor readings, video feeds, and communication data. In this domain, process mining holds the potential for extracting behavioral patterns, identifying anomalies, and evaluating performance metrics in robotic systems. However, its application remains largely unexplored due to the fine-grained, multimodal nature of robotic data. In this work, we investigate how robotic data should be processed to enable process mining applications. We explore activity recognition techniques to transition from robotic fine-grained data to high-level activities, applying Conditional Random Fields to sensor data and fine-tuning the Florence-2 model for video data. Furthermore, we outline key challenges and opportunities in preparing robotic data for process mining, laying the groundwork for future research on process mining-based analysis of robotic systems.