Abstract <p>Accurate neonatal pain assessment is critical for clinical care but remains challenging due to subtle non-verbal expressions and limited labeled data. We introduce SSS-TT (Self-Supervised Sequential Spatio-Temporal Transformers with Cross-Attention Fusion), a novel framework addressing three key limitations: (1) data scarcity through masked autoencoder (MAE) pretraining that reduces annotation requirements by 75% while improving accuracy by 4.9%, (2) inadequate temporal modeling through a cascaded Vision Transformer and Temporal Convolutional Network architecture capturing pain dynamics over 1-second windows, and (3) incomplete multimodal data via cross-attention fusion that maintains 82.9% accuracy with RGB-only input versus 74.2% for robustly-trained static concatenation baselines (trained with identical modality-dropout augmentation). Using ordinal regression with CORAL loss, SSS-TT predicts pain intensity levels (0–3) aligned with clinical NIPS scoring. Evaluated on the iCOPE dataset (1,000 infants, 5-fold cross-validation with strict subject-level splitting to prevent data leakage), our method achieves 84.6% ± 0.7% accuracy and quadratic weighted kappa of 0.82—comparable to the inter-clinician agreement range reported in the literature (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\kappa =0.65\)</EquationSource> </InlineEquation>–0.85 on iCOPE, Fleiss’ <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\kappa =0.81\)</EquationSource> </InlineEquation>)—while significantly outperforming state-of-the-art methods including CNN+LSTM (74.8%), and, under fair comparison conditions (all baselines equipped with the same MAE pretraining and CORAL loss), ViViT +MAE+CORAL (80.4%) and TimeSformer+MAE+CORAL (81.2%). The model demonstrates robustness under clinical degradations with real-time inference (6.7 FPS, 0.15s latency per clip), enabling practical deployment for continuous NICU monitoring. Scenario-based cost-benefit analysis under explicit assumptions suggests 3.4<InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(\times\)</EquationSource> </InlineEquation>–11.9<InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(\times\)</EquationSource> </InlineEquation> first-year ROI depending on healthcare setting and nurse-to-patient ratio. This work advances label-efficient medical AI and multimodal fusion for real-world clinical translation.</p> Graphical abstract <p></p>

SSS-TT: Self-supervised sequential spatio-temporal transformers with adaptive multimodal fusion for automated neonatal pain assessment

期刊论文
发表日期：

2026年4月23日

查看全文

Oussama El Othmani,
Riadh Ouersighni

摘要

Abstract

Accurate neonatal pain assessment is critical for clinical care but remains challenging due to subtle non-verbal expressions and limited labeled data. We introduce SSS-TT (Self-Supervised Sequential Spatio-Temporal Transformers with Cross-Attention Fusion), a novel framework addressing three key limitations: (1) data scarcity through masked autoencoder (MAE) pretraining that reduces annotation requirements by 75% while improving accuracy by 4.9%, (2) inadequate temporal modeling through a cascaded Vision Transformer and Temporal Convolutional Network architecture capturing pain dynamics over 1-second windows, and (3) incomplete multimodal data via cross-attention fusion that maintains 82.9% accuracy with RGB-only input versus 74.2% for robustly-trained static concatenation baselines (trained with identical modality-dropout augmentation). Using ordinal regression with CORAL loss, SSS-TT predicts pain intensity levels (0–3) aligned with clinical NIPS scoring. Evaluated on the iCOPE dataset (1,000 infants, 5-fold cross-validation with strict subject-level splitting to prevent data leakage), our method achieves 84.6% ± 0.7% accuracy and quadratic weighted kappa of 0.82—comparable to the inter-clinician agreement range reported in the literature ( \(\kappa =0.65\) –0.85 on iCOPE, Fleiss’ \(\kappa =0.81\) )—while significantly outperforming state-of-the-art methods including CNN+LSTM (74.8%), and, under fair comparison conditions (all baselines equipped with the same MAE pretraining and CORAL loss), ViViT +MAE+CORAL (80.4%) and TimeSformer+MAE+CORAL (81.2%). The model demonstrates robustness under clinical degradations with real-time inference (6.7 FPS, 0.15s latency per clip), enabling practical deployment for continuous NICU monitoring. Scenario-based cost-benefit analysis under explicit assumptions suggests 3.4 \(\times\) –11.9 \(\times\) first-year ROI depending on healthcare setting and nurse-to-patient ratio. This work advances label-efficient medical AI and multimodal fusion for real-world clinical translation.

Graphical abstract