MPAF-SECL: Multi-path Adaptive Fusion with Semantic-Enhanced Cross-Modal Learning for Skeleton-Based Action Recognition
摘要
Skeleton-based action recognition faces critical challenges including Graph Convolutional Networks’ over-smoothing issues and the semantic limitations of raw skeleton coordinates, which restrict existing methods’ ability to capture comprehensive spatial-temporal dependencies and leverage high-level semantic information for complex action understanding. To address these limitations, we propose MPAF-SECL, a framework that synergistically integrates multi-path adaptive fusion with semantic-enhanced cross-modal learning. Our Multi-path Adaptive Fusion (MPAF) module orchestrates three complementary neural pathways: Graph Convolutional Networks for global structural modeling, Part-Aware Adaptive Attention (PAAA) for fine-grained local feature extraction, and Temporal Convolutional Networks for dynamic pattern capture, effectively mitigating over-smoothing while enabling comprehensive spatial-temporal exploitation across different granularities. Furthermore, we introduce a Semantic-Enhanced Cross-Modal Learning (SECL) module that harnesses pre-trained Large Multimodal Models to provide rich semantic priors, achieving precise cross-modal alignment through bidirectional contrastive learning between skeleton features and textual semantics to substantially enhance semantic understanding. Additionally, our Quad-Stream Adaptive Weighting (QSAW) strategy automatically optimizes fusion coefficients for joint, bone, joint velocity, and bone velocity streams through learnable optimization, maximizing ensemble performance without manual weight assignment. Extensive experiments demonstrate that the MPAF-SECL achieves state-of-the-art performance on multiple benchmark datasets, validating the effectiveness of our multi-path fusion and semantic enhancement strategies.