Multi-stage Interactive Audio-Visual Fusion for Parkinson’s Disease Assisted Diagnosis
摘要
Parkinson’s disease (PD) is a prevalent neurodegenerative disorder with no known cure. Early detection and assisted diagnosis play a crucial role in slowing disease progression, optimizing treatment strategies, and improving patients’ quality of life. This study proposes a multi-stage interactive audio-visual fusion method for PD assisted diagnosis (MSINet). First, a multi-scale dynamic-static feature extraction module is designed. Optical flow is used to capture dynamic facial muscle movements, while image sequences are used to extract static facial expression features. Multi-scale convolution is integrated to enhance the modeling capability for facial motion abnormalities in PD patients. Second, a channel-spatial attention mechanism is introduced in the audio feature extraction network to improve the model’s focus on key speech features. In addition, a dynamic-static feature fusion module and a global-local attention-based audio-visual fusion module are proposed to enable deep interactive integration of facial dynamic features, static features, and speech features. Finally, a progressive multi-stage interactive fusion strategy is constructed, in which fusion modules are embedded at each stage of audio-visual feature extraction. This design facilitates hierarchical cross-modal information exchange and propagates the fused representations to subsequent stages, thereby enhancing deep semantic alignment and complementary representation between modalities. Experimental results on the Chinese PD audio-visual dataset (CPD-AVD) dataset demonstrate that the proposed method achieves significant performance advantages in the assisted diagnosis of Parkinson’s disease.