<p>Skeleton-based action recognition using spatial–temporal graph convolutional networks (ST-GCNs) can maintain performance under heavy parameter pruning, but accuracy degrades when the retained parameter ratio shrinks to 5% or 1%. This work quantitatively analyzes temporal selectivity degradation in sparse temporal branches and identifies it as an important bottleneck under highly sparse and extreme sparsity settings. We propose a lightweight part-motion multi-scale temporal controller (PMM-TC) to recalibrate sparse temporal branches via part-aware state and motion descriptors and multi-scale temporal convolution. Experiments on NTU RGB+D 60 and NTU RGB+D 120 show that PMM-TC improves NTU RGB+D 60 XSub Top-1 accuracy by 2.9 percentage points at <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\mathcal {S}=0.95$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi mathvariant="script">S</mi> <mo>=</mo> <mn>0.95</mn> </mrow> </math></EquationSource> </InlineEquation> and 2.5 percentage points at <InlineEquation ID="IEq2"> <EquationSource Format="TEX">$\mathcal {S}=0.99$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi mathvariant="script">S</mi> <mo>=</mo> <mn>0.99</mn> </mrow> </math></EquationSource> </InlineEquation> with negligible extra computation. Additional validation on NW-UCLA further shows that PMM-TC improves the sparse ST-GCN baseline by 2.3 and 2.9 points at <InlineEquation ID="IEq3"> <EquationSource Format="TEX">$\mathcal {S}=0.95$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi mathvariant="script">S</mi> <mo>=</mo> <mn>0.95</mn> </mrow> </math></EquationSource> </InlineEquation> and <InlineEquation ID="IEq4"> <EquationSource Format="TEX">$\mathcal {S}=0.99$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi mathvariant="script">S</mi> <mo>=</mo> <mn>0.99</mn> </mrow> </math></EquationSource> </InlineEquation>, respectively, suggesting that the proposed part-aware temporal recalibration is not limited to the NTU 25-joint skeleton layout. Stronger-backbone experiments on sparse CTR-GCN further show that PMM-TC consistently improves recognition accuracy at the highly sparse and extreme sparsity settings, indicating that the proposed recalibration is not limited to the sparse ST-GCN baseline. The method partially restores the measured temporal selectivity of sparse temporal branches while preserving efficiency for deployment.</p>

Lightweight multi-scale temporal recalibration for highly sparse skeleton-based action recognition

期刊论文
发表日期：

2026年8月4日

查看全文

Lejia Xu,
Xuehong Cui

摘要

Skeleton-based action recognition using spatial–temporal graph convolutional networks (ST-GCNs) can maintain performance under heavy parameter pruning, but accuracy degrades when the retained parameter ratio shrinks to 5% or 1%. This work quantitatively analyzes temporal selectivity degradation in sparse temporal branches and identifies it as an important bottleneck under highly sparse and extreme sparsity settings. We propose a lightweight part-motion multi-scale temporal controller (PMM-TC) to recalibrate sparse temporal branches via part-aware state and motion descriptors and multi-scale temporal convolution. Experiments on NTU RGB+D 60 and NTU RGB+D 120 show that PMM-TC improves NTU RGB+D 60 XSub Top-1 accuracy by 2.9 percentage points at $\mathcal {S}=0.95$ $S = 0.95$ and 2.5 percentage points at $\mathcal {S}=0.99$ $S = 0.99$ with negligible extra computation. Additional validation on NW-UCLA further shows that PMM-TC improves the sparse ST-GCN baseline by 2.3 and 2.9 points at $\mathcal {S}=0.95$ $S = 0.95$ and $\mathcal {S}=0.99$ $S = 0.99$ , respectively, suggesting that the proposed part-aware temporal recalibration is not limited to the NTU 25-joint skeleton layout. Stronger-backbone experiments on sparse CTR-GCN further show that PMM-TC consistently improves recognition accuracy at the highly sparse and extreme sparsity settings, indicating that the proposed recalibration is not limited to the sparse ST-GCN baseline. The method partially restores the measured temporal selectivity of sparse temporal branches while preserving efficiency for deployment.