This paper introduces a robust multimodal system for predicting Parkinson’s Disease (PD) by meshing together three different data modalities: audio, image, and video. The multimodal system utilizes a different pipeline to process the data independently of one another with state-of-the art deep learning models. Audio data is processed using classical audio feature extraction and classified with ensemble models. Image data is processed with the Xception CNN for feature extraction and classified with an XGBoost classifier. Video data is processed by frames using YOLO object detection models. The pipelines are collated together into a single web application built in Flask. This system is assessed using datasets that are available publicly, demonstrating systematically high performance across the data modalities, approximately 86.7% accuracy based on audio data, approximately 91.2% accuracy based on images, peak accuracy at 94.9% for a separate tabular voice feature dataset. The detector based on video has achieved exceptional accuracy by mean Average Precision (mAP)approximately 0.995. The results indicate that a multimodal approach produces a more trustworthy, more precise, and more effective approach for decision-support in early diagnosis of PD.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Parkinson’s Disease Detection Using Deep Learning

  • H. Poornima,
  • Maria Saji,
  • M. Vidyashree,
  • J. Srujan,
  • Balaraju Kushal Verma

摘要

This paper introduces a robust multimodal system for predicting Parkinson’s Disease (PD) by meshing together three different data modalities: audio, image, and video. The multimodal system utilizes a different pipeline to process the data independently of one another with state-of-the art deep learning models. Audio data is processed using classical audio feature extraction and classified with ensemble models. Image data is processed with the Xception CNN for feature extraction and classified with an XGBoost classifier. Video data is processed by frames using YOLO object detection models. The pipelines are collated together into a single web application built in Flask. This system is assessed using datasets that are available publicly, demonstrating systematically high performance across the data modalities, approximately 86.7% accuracy based on audio data, approximately 91.2% accuracy based on images, peak accuracy at 94.9% for a separate tabular voice feature dataset. The detector based on video has achieved exceptional accuracy by mean Average Precision (mAP)approximately 0.995. The results indicate that a multimodal approach produces a more trustworthy, more precise, and more effective approach for decision-support in early diagnosis of PD.